[Mlir-commits] [flang] [mlir] [flang][OpenACC] Changed memory management for private POINTERs. (PR #219052)
Slava Zakharin
llvmlistbot at llvm.org
Wed Aug 26 14:46:09 PDT 2026
https://github.com/vzakhari created https://github.com/llvm/llvm-project/pull/219052
The motivating example:
```
!$acc parallel loop private(ptr) present(target_data)
do i = 1, 128
ptr => target_data(i)%array
ptr(...) = ...
```
The OpenACC privatization recipe was allocating private data
for `ptr`, storing it in `ptr` private descriptor and then
trying to deallocate `ptr->base_addr` at the end of the region.
`ptr->base_addr` is pointing to `target_data(i)%array` that is allocated
in device global memory and it is illegal to `free()` it in the device code.
The `destroy` part of the recipe can only `free()` the private data
that was originally allocated in the `init` part.
This approach works well for the two most common case.
Case 1: `ptr` is originally unassociated and it is used to point
to some target data (the example above).
Case 2: `ptr` is originally associated and it is used as a normal
private variable for storing something into it without reassociating
`ptr` to some other data.
This change, on the other hand, breaks the following case:
```
allocate(ptr(10))
!$acc parallel private(ptr)...
deallocate(ptr)
ptr => ...
```
The private memory for `ptr` data may be allocated from the host
using CUDA runtime or driver API, and it is illegal to deallocate
it via `free()` which flang-rt's implementation is using.
Moreover, flang-rt's implementation of `DEALLOCATE` assumes
there is an 8-byte footer after the data payload that it uses
to verify whether a `POINTER` deallocation is valid. OpenACC
privatization recipe uses `fir.allocmem` that does not create
the footer, so flang-rt may read past the allocated data
and crash the kernel.
The reference compiler implements `DEALLOCATE` as `free()`, so
the compute-sanitizer reports invalid `free()` errors, though,
w/o the sanitizer the kernel silently passes.
Flang's implementation either fails in `DEALLOCATE` runtime with
`DEALLOCATE of a pointer that is not the whole content of a pointer ALLOCATE`
error or crashes the kernel due to accessing past the allocated memory.
OpenACC spec does not explicitly specify what should happen in this case,
so I suggest that Flang's implementation always reports a runtime error
when program tries to deallocate the private data. This is left for
future improvement.
Assisted-by: Cursor
>From 1990de39bf05aa5827924cfc0da8b2c10fb2aadc Mon Sep 17 00:00:00 2001
From: Slava Zakharin <szakharin at nvidia.com>
Date: Wed, 26 Aug 2026 14:01:34 -0700
Subject: [PATCH] [flang][OpenACC] Changed memory management for private
POINTERs.
The motivating example:
```
!$acc parallel loop private(ptr) present(target_data)
do i = 1, 128
ptr => target_data(i)%array
ptr(...) = ...
```
The OpenACC privatization recipe was allocating private data
for `ptr`, storing it in `ptr` private descriptor and then
trying to deallocate `ptr->base_addr` at the end of the region.
`ptr->base_addr` is pointing to `target_data(i)%array` that is allocated
in device global memory and it is illegal to `free()` it in the device code.
The `destroy` part of the recipe can only `free()` the private data
that was originally allocated in the `init` part.
This approach works well for the two most common case.
Case 1: `ptr` is originally unassociated and it is used to point
to some target data (the example above).
Case 2: `ptr` is originally associated and it is used as a normal
private variable for storing something into it without reassociating
`ptr` to some other data.
This change, on the other hand, breaks the following case:
```
allocate(ptr(10))
!$acc parallel private(ptr)...
deallocate(ptr)
ptr => ...
```
The private memory for `ptr` data may be allocated from the host
using CUDA runtime or driver API, and it is illegal to deallocate
it via `free()` which flang-rt's implementation is using.
Moreover, flang-rt's implementation of `DEALLOCATE` assumes
there is an 8-byte footer after the data payload that it uses
to verify whether a `POINTER` deallocation is valid. OpenACC
privatization recipe uses `fir.allocmem` that does not create
the footer, so flang-rt may read past the allocated data
and crash the kernel.
The reference compiler implements `DEALLOCATE` as `free()`, so
the compute-sanitizer reports invalid `free()` errors, though,
w/o the sanitizer the kernel silently passes.
Flang's implementation either fails in `DEALLOCATE` runtime with
`DEALLOCATE of a pointer that is not the whole content of a pointer ALLOCATE`
error or crashes the kernel due to accessing past the allocated memory.
OpenACC spec does not explicitly specify what should happen in this case,
so I suggest that Flang's implementation always reports a runtime error
when program tries to deallocate the private data. This is left for
future improvement.
Assisted-by: Cursor
---
.../Support/FIROpenACCTypeInterfaces.h | 14 +--
.../Support/FIROpenACCTypeInterfaces.cpp | 104 ++++++++++++++----
.../OpenACC/Support/FIROpenACCUtils.cpp | 22 +++-
.../Fir/OpenACC/recipe-populate-private.mlir | 43 ++++++--
flang/test/Lower/OpenACC/acc-private.f90 | 22 ++--
flang/test/Lower/OpenACC/acc-reduction.f90 | 62 ++++++++---
...cc-recipe-materialization-firstprivate.fir | 43 ++++++++
.../acc-recipe-materialization-private.fir | 35 ++++++
.../mlir/Dialect/OpenACC/OpenACCOps.td | 14 ++-
.../Dialect/OpenACC/OpenACCTypeInterfaces.td | 17 ++-
.../mlir/Dialect/OpenACC/OpenACCUtilsLoop.h | 7 +-
mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp | 48 +++++---
.../Transforms/ACCRecipeMaterialization.cpp | 35 +++---
.../OpenACC/Utils/OpenACCUtilsLoop.cpp | 3 +-
.../Dialect/OpenACC/OpenACCUtilsLoopTest.cpp | 8 +-
15 files changed, 362 insertions(+), 115 deletions(-)
diff --git a/flang/include/flang/Optimizer/OpenACC/Support/FIROpenACCTypeInterfaces.h b/flang/include/flang/Optimizer/OpenACC/Support/FIROpenACCTypeInterfaces.h
index ae47805a1af42..4e4e0d5f2c6f2 100644
--- a/flang/include/flang/Optimizer/OpenACC/Support/FIROpenACCTypeInterfaces.h
+++ b/flang/include/flang/Optimizer/OpenACC/Support/FIROpenACCTypeInterfaces.h
@@ -91,16 +91,16 @@ struct OpenACCMappableModel
genPrivateVariableInfo(mlir::Type type,
mlir::TypedValue<mlir::acc::MappableType> var) const;
- mlir::Value generatePrivateInit(mlir::Type type, mlir::OpBuilder &builder,
- mlir::Location loc,
- mlir::TypedValue<mlir::acc::MappableType> var,
- llvm::StringRef varName,
- mlir::ValueRange extents, mlir::Value initVal,
- mlir::acc::VariableInfoAttr varInfo,
- bool &needsDestroy) const;
+ mlir::Value generatePrivateInit(
+ mlir::Type type, mlir::OpBuilder &builder, mlir::Location loc,
+ mlir::TypedValue<mlir::acc::MappableType> var, llvm::StringRef varName,
+ mlir::ValueRange extents, mlir::Value initVal,
+ mlir::acc::VariableInfoAttr varInfo, bool &needsDestroy,
+ llvm::SmallVectorImpl<mlir::Value> &destroyValues) const;
bool generatePrivateDestroy(mlir::Type type, mlir::OpBuilder &builder,
mlir::Location loc, mlir::Value privatized,
+ mlir::ValueRange destroyValues,
mlir::ValueRange bounds,
mlir::acc::VariableInfoAttr varInfo) const;
diff --git a/flang/lib/Optimizer/OpenACC/Support/FIROpenACCTypeInterfaces.cpp b/flang/lib/Optimizer/OpenACC/Support/FIROpenACCTypeInterfaces.cpp
index e7e522e8e1570..0580eedcac51d 100644
--- a/flang/lib/Optimizer/OpenACC/Support/FIROpenACCTypeInterfaces.cpp
+++ b/flang/lib/Optimizer/OpenACC/Support/FIROpenACCTypeInterfaces.cpp
@@ -741,13 +741,22 @@ mlir::Value OpenACCMappableModel<Ty>::generatePrivateInit(
mlir::Type type, mlir::OpBuilder &mlirBuilder, mlir::Location loc,
mlir::TypedValue<mlir::acc::MappableType> var, llvm::StringRef varName,
mlir::ValueRange bounds, mlir::Value initVal,
- mlir::acc::VariableInfoAttr varInfo, bool &needsDestroy) const {
+ mlir::acc::VariableInfoAttr varInfo, bool &needsDestroy,
+ llvm::SmallVectorImpl<mlir::Value> &destroyValues) const {
mlir::ModuleOp mod = mlirBuilder.getInsertionBlock()
->getParent()
->getParentOfType<mlir::ModuleOp>();
assert(mod && "failed to retrieve ModuleOp");
fir::FirOpBuilder builder(mlirBuilder, mod);
+ hlfir::Entity inputVar = hlfir::Entity{var};
+ bool preservePointerAllocation =
+ fir::isPointerType(inputVar.getType()) && bounds.empty();
+ mlir::Type pointerAllocationType;
+ if (preservePointerAllocation)
+ pointerAllocationType =
+ fir::HeapType::get(inputVar.getElementOrSequenceType());
+
// When variable is optional: use fir.is_present to check. When non-optional,
// skip the conditional to avoid unnecessary branches.
std::optional<fir::IfOp> optIfOp;
@@ -759,13 +768,15 @@ mlir::Value OpenACCMappableModel<Ty>::generatePrivateInit(
if (mayBeOptional) {
mlir::Value cond =
fir::IsPresentOp::create(builder, loc, builder.getI1Type(), var);
- optIfOp = fir::IfOp::create(builder, loc, mlir::TypeRange{type}, cond,
+ llvm::SmallVector<mlir::Type> resultTypes{type};
+ if (preservePointerAllocation)
+ resultTypes.push_back(pointerAllocationType);
+ optIfOp = fir::IfOp::create(builder, loc, resultTypes, cond,
/*withElseRegion=*/true);
builder.setInsertionPointToStart(&optIfOp->getThenRegion().front());
}
}
- hlfir::Entity inputVar = hlfir::Entity{var};
if (inputVar.isPolymorphic())
TODO(loc, "OpenACC: polymorphic variable privatization");
if (auto recType =
@@ -833,6 +844,7 @@ mlir::Value OpenACCMappableModel<Ty>::generatePrivateInit(
mlir::acc::VarNameAttr::get(builder.getContext(),
mlir::acc::getVarNamePlaceholder()));
mlir::Value alloc;
+ mlir::Value pointerAllocation;
if (fir::hasDynamicSize(baseType) ||
(isPointerOrAllocatable && bounds.empty())) {
// Note: heap allocation is forced for whole pointers/allocatable so that
@@ -842,9 +854,32 @@ mlir::Value OpenACCMappableModel<Ty>::generatePrivateInit(
// array POINTER and ALLOCATABLE always have dynamic size. Constant sections
// of POINTER/ALLOCATABLE can use alloca since only part of the data is
// privatized (it makes no sense to deallocate them).
- alloc = builder.createHeapTemporary(loc, baseType, varName, tempExtents,
- typeParams, {placeholderAttr});
+ if (preservePointerAllocation) {
+ mlir::Value originalAddr =
+ hlfir::genVariableRawAddress(loc, builder, inputVar);
+ mlir::Value isAssociated = builder.genIsNotNullAddr(loc, originalAddr);
+ auto ifOp = fir::IfOp::create(
+ builder, loc, mlir::TypeRange{pointerAllocationType}, isAssociated,
+ /*withElseRegion=*/true);
+ builder.setInsertionPointToStart(&ifOp.getThenRegion().front());
+ mlir::Value privateAllocation = builder.createHeapTemporary(
+ loc, baseType, varName, tempExtents, typeParams, {placeholderAttr});
+ fir::ResultOp::create(builder, loc, privateAllocation);
+ builder.setInsertionPointToStart(&ifOp.getElseRegion().front());
+ fir::ResultOp::create(
+ builder, loc, builder.createNullConstant(loc, pointerAllocationType));
+ builder.setInsertionPointAfter(ifOp);
+ alloc = ifOp.getResult(0);
+ } else {
+ alloc = builder.createHeapTemporary(loc, baseType, varName, tempExtents,
+ typeParams, {placeholderAttr});
+ }
needsDestroy = true;
+ if (preservePointerAllocation) {
+ assert(alloc.getType() == pointerAllocationType &&
+ "unexpected private pointer allocation type");
+ pointerAllocation = alloc;
+ }
} else {
alloc = builder.createTemporary(loc, baseType, varName, tempExtents,
typeParams, {placeholderAttr});
@@ -946,11 +981,22 @@ mlir::Value OpenACCMappableModel<Ty>::generatePrivateInit(
}
if (mayBeOptional) {
- fir::ResultOp::create(builder, loc, retVal);
+ llvm::SmallVector<mlir::Value> thenResults{retVal};
+ if (preservePointerAllocation)
+ thenResults.push_back(pointerAllocation);
+ fir::ResultOp::create(builder, loc, thenResults);
builder.setInsertionPointToStart(&optIfOp->getElseRegion().front());
mlir::Value absent = fir::AbsentOp::create(builder, loc, type);
- fir::ResultOp::create(builder, loc, absent);
+ llvm::SmallVector<mlir::Value> elseResults{absent};
+ if (preservePointerAllocation)
+ elseResults.push_back(
+ builder.createNullConstant(loc, pointerAllocationType));
+ fir::ResultOp::create(builder, loc, elseResults);
retVal = optIfOp->getResult(0);
+ if (preservePointerAllocation)
+ destroyValues.push_back(optIfOp->getResult(1));
+ } else if (preservePointerAllocation) {
+ destroyValues.push_back(pointerAllocation);
}
return retVal;
@@ -961,27 +1007,31 @@ OpenACCMappableModel<fir::BaseBoxType>::generatePrivateInit(
mlir::Type type, mlir::OpBuilder &builder, mlir::Location loc,
mlir::TypedValue<mlir::acc::MappableType> var, llvm::StringRef varName,
mlir::ValueRange extents, mlir::Value initVal,
- mlir::acc::VariableInfoAttr varInfo, bool &needsDestroy) const;
+ mlir::acc::VariableInfoAttr varInfo, bool &needsDestroy,
+ llvm::SmallVectorImpl<mlir::Value> &destroyValues) const;
template mlir::Value
OpenACCMappableModel<fir::ReferenceType>::generatePrivateInit(
mlir::Type type, mlir::OpBuilder &builder, mlir::Location loc,
mlir::TypedValue<mlir::acc::MappableType> var, llvm::StringRef varName,
mlir::ValueRange extents, mlir::Value initVal,
- mlir::acc::VariableInfoAttr varInfo, bool &needsDestroy) const;
+ mlir::acc::VariableInfoAttr varInfo, bool &needsDestroy,
+ llvm::SmallVectorImpl<mlir::Value> &destroyValues) const;
template mlir::Value OpenACCMappableModel<fir::HeapType>::generatePrivateInit(
mlir::Type type, mlir::OpBuilder &builder, mlir::Location loc,
mlir::TypedValue<mlir::acc::MappableType> var, llvm::StringRef varName,
mlir::ValueRange extents, mlir::Value initVal,
- mlir::acc::VariableInfoAttr varInfo, bool &needsDestroy) const;
+ mlir::acc::VariableInfoAttr varInfo, bool &needsDestroy,
+ llvm::SmallVectorImpl<mlir::Value> &destroyValues) const;
template mlir::Value
OpenACCMappableModel<fir::PointerType>::generatePrivateInit(
mlir::Type type, mlir::OpBuilder &builder, mlir::Location loc,
mlir::TypedValue<mlir::acc::MappableType> var, llvm::StringRef varName,
mlir::ValueRange extents, mlir::Value initVal,
- mlir::acc::VariableInfoAttr varInfo, bool &needsDestroy) const;
+ mlir::acc::VariableInfoAttr varInfo, bool &needsDestroy,
+ llvm::SmallVectorImpl<mlir::Value> &destroyValues) const;
template <typename Ty>
bool OpenACCMappableModel<Ty>::generateCopy(
@@ -1242,13 +1292,25 @@ template bool OpenACCMappableModel<fir::HeapType>::generateCombiner(
template <typename Ty>
bool OpenACCMappableModel<Ty>::generatePrivateDestroy(
mlir::Type type, mlir::OpBuilder &mlirBuilder, mlir::Location loc,
- mlir::Value privatized, mlir::ValueRange bounds,
- mlir::acc::VariableInfoAttr varInfo) const {
+ mlir::Value privatized, mlir::ValueRange destroyValues,
+ mlir::ValueRange bounds, mlir::acc::VariableInfoAttr varInfo) const {
hlfir::Entity inputVar = hlfir::Entity{privatized};
mlir::ModuleOp mod =
mlirBuilder.getBlock()->getParent()->getParentOfType<mlir::ModuleOp>();
assert(mod && "failed to retrieve parent module");
fir::FirOpBuilder builder(mlirBuilder, mod);
+ if (!destroyValues.empty()) {
+ assert(destroyValues.size() == 1 &&
+ "expected one private pointer allocation to destroy");
+ mlir::Value privateAllocation = destroyValues.front();
+ mlir::Value isAllocated = builder.genIsNotNullAddr(loc, privateAllocation);
+ auto ifOp =
+ fir::IfOp::create(builder, loc, isAllocated, /*withElseRegion=*/false);
+ builder.setInsertionPointToStart(&ifOp.getThenRegion().front());
+ fir::FreeMemOp::create(builder, loc, privateAllocation);
+ builder.setInsertionPointAfter(ifOp);
+ return true;
+ }
auto genFreeRawAddress = [&](hlfir::Entity entity) {
mlir::Value addr = hlfir::genVariableRawAddress(loc, builder, entity);
mlir::Type heapType =
@@ -1278,20 +1340,20 @@ bool OpenACCMappableModel<Ty>::generatePrivateDestroy(
template bool OpenACCMappableModel<fir::BaseBoxType>::generatePrivateDestroy(
mlir::Type type, mlir::OpBuilder &builder, mlir::Location loc,
- mlir::Value privatized, mlir::ValueRange bounds,
- mlir::acc::VariableInfoAttr varInfo) const;
+ mlir::Value privatized, mlir::ValueRange destroyValues,
+ mlir::ValueRange bounds, mlir::acc::VariableInfoAttr varInfo) const;
template bool OpenACCMappableModel<fir::ReferenceType>::generatePrivateDestroy(
mlir::Type type, mlir::OpBuilder &builder, mlir::Location loc,
- mlir::Value privatized, mlir::ValueRange bounds,
- mlir::acc::VariableInfoAttr varInfo) const;
+ mlir::Value privatized, mlir::ValueRange destroyValues,
+ mlir::ValueRange bounds, mlir::acc::VariableInfoAttr varInfo) const;
template bool OpenACCMappableModel<fir::HeapType>::generatePrivateDestroy(
mlir::Type type, mlir::OpBuilder &builder, mlir::Location loc,
- mlir::Value privatized, mlir::ValueRange bounds,
- mlir::acc::VariableInfoAttr varInfo) const;
+ mlir::Value privatized, mlir::ValueRange destroyValues,
+ mlir::ValueRange bounds, mlir::acc::VariableInfoAttr varInfo) const;
template bool OpenACCMappableModel<fir::PointerType>::generatePrivateDestroy(
mlir::Type type, mlir::OpBuilder &builder, mlir::Location loc,
- mlir::Value privatized, mlir::ValueRange bounds,
- mlir::acc::VariableInfoAttr varInfo) const;
+ mlir::Value privatized, mlir::ValueRange destroyValues,
+ mlir::ValueRange bounds, mlir::acc::VariableInfoAttr varInfo) const;
template <typename Ty>
mlir::Value OpenACCPointerLikeModel<Ty>::genAllocate(
diff --git a/flang/lib/Optimizer/OpenACC/Support/FIROpenACCUtils.cpp b/flang/lib/Optimizer/OpenACC/Support/FIROpenACCUtils.cpp
index 4417b769fa314..77c9f2b2b9700 100644
--- a/flang/lib/Optimizer/OpenACC/Support/FIROpenACCUtils.cpp
+++ b/flang/lib/Optimizer/OpenACC/Support/FIROpenACCUtils.cpp
@@ -475,13 +475,16 @@ static RecipeOp genRecipeOp(
"Expected that all variable types are considered mappable");
auto initArg = mlir::cast<MappableValue>(initBlock->getArgument(0));
bool needsDestroy = false;
+ llvm::SmallVector<mlir::Value> destroyValues;
llvm::SmallVector<mlir::Value> initBounds =
getRecipeBounds(builder, loc, dataOperationBounds,
initBlock->getArguments().drop_front(1));
mlir::Value retVal = mappableTy.generatePrivateInit(
builder, loc, initArg, initName, initBounds, initValue, varInfo,
- needsDestroy);
- mlir::acc::YieldOp::create(builder, loc, retVal);
+ needsDestroy, destroyValues);
+ llvm::SmallVector<mlir::Value> initResults{retVal};
+ initResults.append(destroyValues);
+ mlir::acc::YieldOp::create(builder, loc, initResults);
// Create destroy region and generate destruction if requested.
if (needsDestroy) {
llvm::SmallVector<mlir::Type> destroyArgsTy;
@@ -491,6 +494,10 @@ static RecipeOp genRecipeOp(
destroyArgsTy.push_back(ty);
destroyArgsLoc.push_back(loc);
destroyArgsLoc.push_back(loc);
+ for (mlir::Value destroyValue : destroyValues) {
+ destroyArgsTy.push_back(destroyValue.getType());
+ destroyArgsLoc.push_back(loc);
+ }
// Append bounds arguments (if any) in the same order as init region
if (argsTy.size() > 1) {
destroyArgsTy.append(argsTy.begin() + 1, argsTy.end());
@@ -502,11 +509,14 @@ static RecipeOp genRecipeOp(
destroyArgsTy, destroyArgsLoc);
builder.setInsertionPointToEnd(destroyBlock);
- llvm::SmallVector<mlir::Value> destroyBounds =
- getRecipeBounds(builder, loc, dataOperationBounds,
- destroyBlock->getArguments().drop_front(2));
+ llvm::SmallVector<mlir::Value> destroyBounds = getRecipeBounds(
+ builder, loc, dataOperationBounds,
+ destroyBlock->getArguments().drop_front(2 + destroyValues.size()));
+ mlir::ValueRange destroyArgs =
+ destroyBlock->getArguments().slice(2, destroyValues.size());
[[maybe_unused]] bool success = mappableTy.generatePrivateDestroy(
- builder, loc, destroyBlock->getArgument(1), destroyBounds, varInfo);
+ builder, loc, destroyBlock->getArgument(1), destroyArgs, destroyBounds,
+ varInfo);
assert(success && "failed to generate destroy region");
mlir::acc::TerminatorOp::create(builder, loc);
}
diff --git a/flang/test/Fir/OpenACC/recipe-populate-private.mlir b/flang/test/Fir/OpenACC/recipe-populate-private.mlir
index f5200f70a3ba8..c5e37db7d450b 100644
--- a/flang/test/Fir/OpenACC/recipe-populate-private.mlir
+++ b/flang/test/Fir/OpenACC/recipe-populate-private.mlir
@@ -115,7 +115,10 @@ func.func @test_derived() {
// CHECK: fir.store %[[EMBOX]] to %[[BOXALLOC]] : !fir.ref<!fir.box<!fir.heap<f64>>>
// CHECK: acc.yield %[[BOXALLOC]] : !fir.ref<!fir.box<!fir.heap<f64>>>
// CHECK: } destroy {
-// CHECK: ^bb0(%{{.*}}: !fir.ref<!fir.box<!fir.heap<f64>>>, %{{.*}}: !fir.ref<!fir.box<!fir.heap<f64>>>):
+// CHECK: ^bb0(%{{.*}}: !fir.ref<!fir.box<!fir.heap<f64>>>, %[[PRIVATE:.*]]: !fir.ref<!fir.box<!fir.heap<f64>>>):
+// CHECK: %[[BOX:.*]] = fir.load %[[PRIVATE]] : !fir.ref<!fir.box<!fir.heap<f64>>>
+// CHECK: %[[ADDR:.*]] = fir.box_addr %[[BOX]] : (!fir.box<!fir.heap<f64>>) -> !fir.heap<f64>
+// CHECK: fir.freemem %[[ADDR]] : !fir.heap<f64>
// CHECK: acc.terminator
// CHECK: }
@@ -131,13 +134,23 @@ func.func @test_box_heap_scalar() {
// Test box type with pointer scalar (needs destroy)
// CHECK: acc.private.recipe @private_box_ptr_scalar : !fir.ref<!fir.box<!fir.ptr<i32>>> init {
// CHECK: ^bb0(%{{.*}}: !fir.ref<!fir.box<!fir.ptr<i32>>>):
-// CHECK: %[[SCALAR:.*]] = fir.allocmem i32
-// CHECK: %[[EMBOX:.*]] = fir.embox %[[SCALAR]] : (!fir.heap<i32>) -> !fir.box<!fir.ptr<i32>>
+// CHECK: %[[PRIVATE_ALLOC:.*]] = fir.if {{.*}} -> (!fir.heap<i32>) {
+// CHECK: %[[SCALAR:.*]] = fir.allocmem i32
+// CHECK: fir.result %[[SCALAR]] : !fir.heap<i32>
+// CHECK: } else {
+// CHECK: %[[NULL_ALLOC:.*]] = fir.zero_bits !fir.heap<i32>
+// CHECK: fir.result %[[NULL_ALLOC]] : !fir.heap<i32>
+// CHECK: }
+// CHECK: %[[EMBOX:.*]] = fir.embox %[[PRIVATE_ALLOC]] : (!fir.heap<i32>) -> !fir.box<!fir.ptr<i32>>
// CHECK: %[[BOXALLOC:.*]] = fir.alloca !fir.box<!fir.ptr<i32>>
// CHECK: fir.store %[[EMBOX]] to %[[BOXALLOC]] : !fir.ref<!fir.box<!fir.ptr<i32>>>
-// CHECK: acc.yield %[[BOXALLOC]] : !fir.ref<!fir.box<!fir.ptr<i32>>>
+// CHECK: acc.yield %[[BOXALLOC]], %[[PRIVATE_ALLOC]] : !fir.ref<!fir.box<!fir.ptr<i32>>>, !fir.heap<i32>
// CHECK: } destroy {
-// CHECK: ^bb0(%{{.*}}: !fir.ref<!fir.box<!fir.ptr<i32>>>, %{{.*}}: !fir.ref<!fir.box<!fir.ptr<i32>>>):
+// CHECK: ^bb0(%{{.*}}: !fir.ref<!fir.box<!fir.ptr<i32>>>, %{{.*}}: !fir.ref<!fir.box<!fir.ptr<i32>>>, %[[PRIVATE_ALLOC:.*]]: !fir.heap<i32>):
+// CHECK-NOT: fir.box_addr
+// CHECK: fir.if {{.*}} {
+// CHECK: fir.freemem %[[PRIVATE_ALLOC]] : !fir.heap<i32>
+// CHECK: }
// CHECK: acc.terminator
// CHECK: }
@@ -156,7 +169,10 @@ func.func @test_box_ptr_scalar() {
// CHECK: %[[BOXALLOC:.*]] = fir.alloca !fir.box<!fir.heap<!fir.array<?xf32>>>
// CHECK: acc.yield %[[BOXALLOC]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>
// CHECK: } destroy {
-// CHECK: ^bb0(%{{.*}}: !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>, %{{.*}}: !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>):
+// CHECK: ^bb0(%{{.*}}: !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>, %[[PRIVATE:.*]]: !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>):
+// CHECK: %[[BOX:.*]] = fir.load %[[PRIVATE]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>
+// CHECK: %[[ADDR:.*]] = fir.box_addr %[[BOX]] : (!fir.box<!fir.heap<!fir.array<?xf32>>>) -> !fir.heap<!fir.array<?xf32>>
+// CHECK: fir.freemem %[[ADDR]] : !fir.heap<!fir.array<?xf32>>
// CHECK: acc.terminator
// CHECK: }
@@ -191,10 +207,21 @@ func.func @test_box_heap_array_2d() {
// Test box type with pointer array (needs destroy)
// CHECK: acc.private.recipe @private_box_ptr_array : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>> init {
// CHECK: ^bb0(%{{.*}}: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>):
+// CHECK: %[[PRIVATE_ALLOC:.*]] = fir.if {{.*}} -> (!fir.heap<!fir.array<?xf32>>) {
+// CHECK: %[[ARRAY:.*]] = fir.allocmem !fir.array<?xf32>
+// CHECK: fir.result %[[ARRAY]] : !fir.heap<!fir.array<?xf32>>
+// CHECK: } else {
+// CHECK: %[[NULL_ALLOC:.*]] = fir.zero_bits !fir.heap<!fir.array<?xf32>>
+// CHECK: fir.result %[[NULL_ALLOC]] : !fir.heap<!fir.array<?xf32>>
+// CHECK: }
// CHECK: %[[BOXALLOC:.*]] = fir.alloca !fir.box<!fir.ptr<!fir.array<?xf32>>>
-// CHECK: acc.yield %[[BOXALLOC]] : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>
+// CHECK: acc.yield %[[BOXALLOC]], %[[PRIVATE_ALLOC]] : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>, !fir.heap<!fir.array<?xf32>>
// CHECK: } destroy {
-// CHECK: ^bb0(%{{.*}}: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>, %{{.*}}: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>):
+// CHECK: ^bb0(%{{.*}}: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>, %{{.*}}: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>, %[[PRIVATE_ALLOC:.*]]: !fir.heap<!fir.array<?xf32>>):
+// CHECK-NOT: fir.box_addr
+// CHECK: fir.if {{.*}} {
+// CHECK: fir.freemem %[[PRIVATE_ALLOC]] : !fir.heap<!fir.array<?xf32>>
+// CHECK: }
// CHECK: acc.terminator
// CHECK: }
diff --git a/flang/test/Lower/OpenACC/acc-private.f90 b/flang/test/Lower/OpenACC/acc-private.f90
index 67709f1e9e7ce..fdebdc2ed6f2a 100644
--- a/flang/test/Lower/OpenACC/acc-private.f90
+++ b/flang/test/Lower/OpenACC/acc-private.f90
@@ -173,23 +173,29 @@
! CHECK: %[[BOX_DIMS_0:.*]]:3 = fir.box_dims %[[LOAD_0]], %[[CONSTANT_0]] : (!fir.box<!fir.ptr<!fir.array<?xi32>>>, index) -> (index, index, index)
! CHECK: %[[SHAPE_0:.*]] = fir.shape %[[BOX_DIMS_0]]#1 : (index) -> !fir.shape<1>
! CHECK: %[[SHAPE_1:.*]] = fir.shape %[[BOX_DIMS_0]]#1 : (index) -> !fir.shape<1>
-! CHECK: %[[ALLOCMEM_0:.*]] = fir.allocmem !fir.array<?xi32>, %[[BOX_DIMS_0]]#1 {acc.var_name = #acc.var_name<"<acc.varname.placeholder>">, bindc_name = "acc.private.init", uniq_name = ""}
+! CHECK: %[[PRIVATE_ALLOC:.*]] = fir.if {{.*}} -> (!fir.heap<!fir.array<?xi32>>) {
+! CHECK: %[[ALLOCMEM_0:.*]] = fir.allocmem !fir.array<?xi32>, %[[BOX_DIMS_0]]#1 {acc.var_name = #acc.var_name<"<acc.varname.placeholder>">, bindc_name = "acc.private.init", uniq_name = ""}
+! CHECK: fir.result %[[ALLOCMEM_0]] : !fir.heap<!fir.array<?xi32>>
+! CHECK: } else {
+! CHECK: %[[NULL_ALLOC:.*]] = fir.zero_bits !fir.heap<!fir.array<?xi32>>
+! CHECK: fir.result %[[NULL_ALLOC]] : !fir.heap<!fir.array<?xi32>>
+! CHECK: }
! CHECK: %[[CONSTANT_1:.*]] = arith.constant 0 : index
! CHECK: %[[BOX_DIMS_1:.*]]:3 = fir.box_dims %[[LOAD_0]], %[[CONSTANT_1]] : (!fir.box<!fir.ptr<!fir.array<?xi32>>>, index) -> (index, index, index)
! CHECK: %[[CONSTANT_2:.*]] = arith.constant 0 : index
! CHECK: %[[BOX_DIMS_2:.*]]:3 = fir.box_dims %[[LOAD_0]], %[[CONSTANT_2]] : (!fir.box<!fir.ptr<!fir.array<?xi32>>>, index) -> (index, index, index)
! CHECK: %[[SHAPE_SHIFT_0:.*]] = fir.shape_shift %[[BOX_DIMS_1]]#0, %[[BOX_DIMS_2]]#1 : (index, index) -> !fir.shapeshift<1>
-! CHECK: %[[EMBOX_0:.*]] = fir.embox %[[ALLOCMEM_0]](%[[SHAPE_SHIFT_0]]) : (!fir.heap<!fir.array<?xi32>>, !fir.shapeshift<1>) -> !fir.box<!fir.ptr<!fir.array<?xi32>>>
+! CHECK: %[[EMBOX_0:.*]] = fir.embox %[[PRIVATE_ALLOC]](%[[SHAPE_SHIFT_0]]) : (!fir.heap<!fir.array<?xi32>>, !fir.shapeshift<1>) -> !fir.box<!fir.ptr<!fir.array<?xi32>>>
! CHECK: %[[ALLOCA_0:.*]] = fir.alloca !fir.box<!fir.ptr<!fir.array<?xi32>>>
! CHECK: fir.store %[[EMBOX_0]] to %[[ALLOCA_0]] : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>
-! CHECK: acc.yield %[[ALLOCA_0]] : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>
+! CHECK: acc.yield %[[ALLOCA_0]], %[[PRIVATE_ALLOC]] : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>, !fir.heap<!fir.array<?xi32>>
! CHECK-LABEL: } destroy {
-! CHECK: ^bb0(%[[VAL_0:.*]]: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>, %[[VAL_1:.*]]: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>):
-! CHECK: %[[LOAD_0:.*]] = fir.load %[[VAL_1]] : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>
-! CHECK: %[[BOX_ADDR_0:.*]] = fir.box_addr %[[LOAD_0]] : (!fir.box<!fir.ptr<!fir.array<?xi32>>>) -> !fir.ptr<!fir.array<?xi32>>
-! CHECK: %[[CONVERT_0:.*]] = fir.convert %[[BOX_ADDR_0]] : (!fir.ptr<!fir.array<?xi32>>) -> !fir.heap<!fir.array<?xi32>>
-! CHECK: fir.freemem %[[CONVERT_0]] : !fir.heap<!fir.array<?xi32>>
+! CHECK: ^bb0(%[[VAL_0:.*]]: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>, %[[VAL_1:.*]]: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>, %[[PRIVATE_ALLOC:.*]]: !fir.heap<!fir.array<?xi32>>):
+! CHECK-NOT: fir.box_addr
+! CHECK: fir.if {{.*}} {
+! CHECK: fir.freemem %[[PRIVATE_ALLOC]] : !fir.heap<!fir.array<?xi32>>
+! CHECK: }
! CHECK: acc.terminator
! CHECK: }
diff --git a/flang/test/Lower/OpenACC/acc-reduction.f90 b/flang/test/Lower/OpenACC/acc-reduction.f90
index f4fc0910e678a..de60d4dfd432f 100644
--- a/flang/test/Lower/OpenACC/acc-reduction.f90
+++ b/flang/test/Lower/OpenACC/acc-reduction.f90
@@ -129,8 +129,19 @@
! CHECK: %[[BOX_DIMS_0:.*]]:3 = fir.box_dims %[[LOAD_0]], %[[CONSTANT_1]] : (!fir.box<!fir.ptr<!fir.array<?xf32>>>, index) -> (index, index, index)
! CHECK: %[[SHAPE_0:.*]] = fir.shape %[[BOX_DIMS_0]]#1 : (index) -> !fir.shape<1>
! CHECK: %[[SHAPE_1:.*]] = fir.shape %[[BOX_DIMS_0]]#1 : (index) -> !fir.shape<1>
-! CHECK: %[[ALLOCMEM_0:.*]] = fir.allocmem !fir.array<?xf32>, %[[BOX_DIMS_0]]#1 {acc.var_name = #acc.var_name<"<acc.varname.placeholder>">, bindc_name = "acc.reduction.init", uniq_name = ""}
-! CHECK: %[[EMBOX_0:.*]] = fir.embox %[[ALLOCMEM_0]](%[[SHAPE_1]]) : (!fir.heap<!fir.array<?xf32>>, !fir.shape<1>) -> !fir.box<!fir.array<?xf32>>
+! CHECK: %[[LOAD_1:.*]] = fir.load %[[VAL_0]] : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>
+! CHECK: %[[BOX_ADDR_0:.*]] = fir.box_addr %[[LOAD_1]] : (!fir.box<!fir.ptr<!fir.array<?xf32>>>) -> !fir.ptr<!fir.array<?xf32>>
+! CHECK: %[[CONVERT_0:.*]] = fir.convert %[[BOX_ADDR_0]] : (!fir.ptr<!fir.array<?xf32>>) -> i64
+! CHECK: %[[CONSTANT_I64:.*]] = arith.constant 0 : i64
+! CHECK: %[[IS_ASSOCIATED:.*]] = arith.cmpi ne, %[[CONVERT_0]], %[[CONSTANT_I64]] : i64
+! CHECK: %[[PRIVATE_ALLOC:.*]] = fir.if %[[IS_ASSOCIATED]] -> (!fir.heap<!fir.array<?xf32>>) {
+! CHECK: %[[ALLOCMEM_0:.*]] = fir.allocmem !fir.array<?xf32>, %[[BOX_DIMS_0]]#1 {acc.var_name = #acc.var_name<"<acc.varname.placeholder>">, bindc_name = "acc.reduction.init", uniq_name = ""}
+! CHECK: fir.result %[[ALLOCMEM_0]] : !fir.heap<!fir.array<?xf32>>
+! CHECK: } else {
+! CHECK: %[[ZERO_BITS:.*]] = fir.zero_bits !fir.heap<!fir.array<?xf32>>
+! CHECK: fir.result %[[ZERO_BITS]] : !fir.heap<!fir.array<?xf32>>
+! CHECK: }
+! CHECK: %[[EMBOX_0:.*]] = fir.embox %[[PRIVATE_ALLOC]](%[[SHAPE_1]]) : (!fir.heap<!fir.array<?xf32>>, !fir.shape<1>) -> !fir.box<!fir.array<?xf32>>
! CHECK: %[[CONSTANT_2:.*]] = arith.constant 0 : index
! CHECK: %[[BOX_DIMS_1:.*]]:3 = fir.box_dims %[[EMBOX_0]], %[[CONSTANT_2]] : (!fir.box<!fir.array<?xf32>>, index) -> (index, index, index)
! CHECK: %[[SHAPE_2:.*]] = fir.shape %[[BOX_DIMS_1]]#1 : (index) -> !fir.shape<1>
@@ -144,10 +155,10 @@
! CHECK: %[[CONSTANT_5:.*]] = arith.constant 0 : index
! CHECK: %[[BOX_DIMS_3:.*]]:3 = fir.box_dims %[[LOAD_0]], %[[CONSTANT_5]] : (!fir.box<!fir.ptr<!fir.array<?xf32>>>, index) -> (index, index, index)
! CHECK: %[[SHAPE_SHIFT_0:.*]] = fir.shape_shift %[[BOX_DIMS_2]]#0, %[[BOX_DIMS_3]]#1 : (index, index) -> !fir.shapeshift<1>
-! CHECK: %[[EMBOX_1:.*]] = fir.embox %[[ALLOCMEM_0]](%[[SHAPE_SHIFT_0]]) : (!fir.heap<!fir.array<?xf32>>, !fir.shapeshift<1>) -> !fir.box<!fir.ptr<!fir.array<?xf32>>>
+! CHECK: %[[EMBOX_1:.*]] = fir.embox %[[PRIVATE_ALLOC]](%[[SHAPE_SHIFT_0]]) : (!fir.heap<!fir.array<?xf32>>, !fir.shapeshift<1>) -> !fir.box<!fir.ptr<!fir.array<?xf32>>>
! CHECK: %[[ALLOCA_0:.*]] = fir.alloca !fir.box<!fir.ptr<!fir.array<?xf32>>>
! CHECK: fir.store %[[EMBOX_1]] to %[[ALLOCA_0]] : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>
-! CHECK: acc.yield %[[ALLOCA_0]] : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>
+! CHECK: acc.yield %[[ALLOCA_0]], %[[PRIVATE_ALLOC]] : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>, !fir.heap<!fir.array<?xf32>>
! CHECK-LABEL: } combiner {
! CHECK: ^bb0(%[[VAL_0:.*]]: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>, %[[VAL_1:.*]]: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>):
@@ -182,11 +193,13 @@
! CHECK: acc.yield %[[VAL_0]] : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>
! CHECK-LABEL: } destroy {
-! CHECK: ^bb0(%[[VAL_0:.*]]: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>, %[[VAL_1:.*]]: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>):
-! CHECK: %[[LOAD_0:.*]] = fir.load %[[VAL_1]] : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>
-! CHECK: %[[BOX_ADDR_0:.*]] = fir.box_addr %[[LOAD_0]] : (!fir.box<!fir.ptr<!fir.array<?xf32>>>) -> !fir.ptr<!fir.array<?xf32>>
-! CHECK: %[[CONVERT_0:.*]] = fir.convert %[[BOX_ADDR_0]] : (!fir.ptr<!fir.array<?xf32>>) -> !fir.heap<!fir.array<?xf32>>
-! CHECK: fir.freemem %[[CONVERT_0]] : !fir.heap<!fir.array<?xf32>>
+! CHECK: ^bb0(%[[VAL_0:.*]]: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>, %[[VAL_1:.*]]: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>, %[[PRIVATE_ALLOC:.*]]: !fir.heap<!fir.array<?xf32>>):
+! CHECK: %[[CONVERT_0:.*]] = fir.convert %[[PRIVATE_ALLOC]] : (!fir.heap<!fir.array<?xf32>>) -> i64
+! CHECK: %[[CONSTANT_0:.*]] = arith.constant 0 : i64
+! CHECK: %[[IS_ALLOCATED:.*]] = arith.cmpi ne, %[[CONVERT_0]], %[[CONSTANT_0]] : i64
+! CHECK: fir.if %[[IS_ALLOCATED]] {
+! CHECK: fir.freemem %[[PRIVATE_ALLOC]] : !fir.heap<!fir.array<?xf32>>
+! CHECK: }
! CHECK: acc.terminator
! CHECK: }
@@ -546,12 +559,23 @@
! CHECK: %[[CONSTANT_0:.*]] = arith.constant 0 : i32
! CHECK: %[[LOAD_0:.*]] = fir.load %[[VAL_0]] : !fir.ref<!fir.box<!fir.ptr<i32>>>
! CHECK: %[[BOX_ADDR_0:.*]] = fir.box_addr %[[LOAD_0]] : (!fir.box<!fir.ptr<i32>>) -> !fir.ptr<i32>
-! CHECK: %[[ALLOCMEM_0:.*]] = fir.allocmem i32 {acc.var_name = #acc.var_name<"<acc.varname.placeholder>">, bindc_name = "acc.reduction.init", uniq_name = ""}
-! CHECK: hlfir.assign %[[CONSTANT_0]] to %[[ALLOCMEM_0]] temporary_lhs : i32, !fir.heap<i32>
-! CHECK: %[[EMBOX_0:.*]] = fir.embox %[[ALLOCMEM_0]] : (!fir.heap<i32>) -> !fir.box<!fir.ptr<i32>>
+! CHECK: %[[LOAD_1:.*]] = fir.load %[[VAL_0]] : !fir.ref<!fir.box<!fir.ptr<i32>>>
+! CHECK: %[[BOX_ADDR_1:.*]] = fir.box_addr %[[LOAD_1]] : (!fir.box<!fir.ptr<i32>>) -> !fir.ptr<i32>
+! CHECK: %[[CONVERT_0:.*]] = fir.convert %[[BOX_ADDR_1]] : (!fir.ptr<i32>) -> i64
+! CHECK: %[[CONSTANT_I64:.*]] = arith.constant 0 : i64
+! CHECK: %[[IS_ASSOCIATED:.*]] = arith.cmpi ne, %[[CONVERT_0]], %[[CONSTANT_I64]] : i64
+! CHECK: %[[PRIVATE_ALLOC:.*]] = fir.if %[[IS_ASSOCIATED]] -> (!fir.heap<i32>) {
+! CHECK: %[[ALLOCMEM_0:.*]] = fir.allocmem i32 {acc.var_name = #acc.var_name<"<acc.varname.placeholder>">, bindc_name = "acc.reduction.init", uniq_name = ""}
+! CHECK: fir.result %[[ALLOCMEM_0]] : !fir.heap<i32>
+! CHECK: } else {
+! CHECK: %[[ZERO_BITS:.*]] = fir.zero_bits !fir.heap<i32>
+! CHECK: fir.result %[[ZERO_BITS]] : !fir.heap<i32>
+! CHECK: }
+! CHECK: hlfir.assign %[[CONSTANT_0]] to %[[PRIVATE_ALLOC]] temporary_lhs : i32, !fir.heap<i32>
+! CHECK: %[[EMBOX_0:.*]] = fir.embox %[[PRIVATE_ALLOC]] : (!fir.heap<i32>) -> !fir.box<!fir.ptr<i32>>
! CHECK: %[[ALLOCA_0:.*]] = fir.alloca !fir.box<!fir.ptr<i32>>
! CHECK: fir.store %[[EMBOX_0]] to %[[ALLOCA_0]] : !fir.ref<!fir.box<!fir.ptr<i32>>>
-! CHECK: acc.yield %[[ALLOCA_0]] : !fir.ref<!fir.box<!fir.ptr<i32>>>
+! CHECK: acc.yield %[[ALLOCA_0]], %[[PRIVATE_ALLOC]] : !fir.ref<!fir.box<!fir.ptr<i32>>>, !fir.heap<i32>
! CHECK-LABEL: } combiner {
! CHECK: ^bb0(%[[VAL_0:.*]]: !fir.ref<!fir.box<!fir.ptr<i32>>>, %[[VAL_1:.*]]: !fir.ref<!fir.box<!fir.ptr<i32>>>):
@@ -563,11 +587,13 @@
! CHECK: acc.yield %[[VAL_0]] : !fir.ref<!fir.box<!fir.ptr<i32>>>
! CHECK-LABEL: } destroy {
-! CHECK: ^bb0(%[[VAL_0:.*]]: !fir.ref<!fir.box<!fir.ptr<i32>>>, %[[VAL_1:.*]]: !fir.ref<!fir.box<!fir.ptr<i32>>>):
-! CHECK: %[[LOAD_0:.*]] = fir.load %[[VAL_1]] : !fir.ref<!fir.box<!fir.ptr<i32>>>
-! CHECK: %[[BOX_ADDR_0:.*]] = fir.box_addr %[[LOAD_0]] : (!fir.box<!fir.ptr<i32>>) -> !fir.ptr<i32>
-! CHECK: %[[CONVERT_0:.*]] = fir.convert %[[BOX_ADDR_0]] : (!fir.ptr<i32>) -> !fir.heap<i32>
-! CHECK: fir.freemem %[[CONVERT_0]] : !fir.heap<i32>
+! CHECK: ^bb0(%[[VAL_0:.*]]: !fir.ref<!fir.box<!fir.ptr<i32>>>, %[[VAL_1:.*]]: !fir.ref<!fir.box<!fir.ptr<i32>>>, %[[PRIVATE_ALLOC:.*]]: !fir.heap<i32>):
+! CHECK: %[[CONVERT_0:.*]] = fir.convert %[[PRIVATE_ALLOC]] : (!fir.heap<i32>) -> i64
+! CHECK: %[[CONSTANT_0:.*]] = arith.constant 0 : i64
+! CHECK: %[[IS_ALLOCATED:.*]] = arith.cmpi ne, %[[CONVERT_0]], %[[CONSTANT_0]] : i64
+! CHECK: fir.if %[[IS_ALLOCATED]] {
+! CHECK: fir.freemem %[[PRIVATE_ALLOC]] : !fir.heap<i32>
+! CHECK: }
! CHECK: acc.terminator
! CHECK: }
diff --git a/flang/test/Transforms/OpenACC/acc-recipe-materialization-firstprivate.fir b/flang/test/Transforms/OpenACC/acc-recipe-materialization-firstprivate.fir
index 12eecb6cab672..1a90ea30cf2d3 100644
--- a/flang/test/Transforms/OpenACC/acc-recipe-materialization-firstprivate.fir
+++ b/flang/test/Transforms/OpenACC/acc-recipe-materialization-firstprivate.fir
@@ -33,6 +33,38 @@ module {
}
return
}
+
+ acc.firstprivate.recipe @firstprivatization_pointer : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>> init {
+ ^bb0(%arg0: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>):
+ %c4 = arith.constant 4 : index
+ %alloc = fir.allocmem !fir.array<?xf32>, %c4
+ %shape = fir.shape %c4 : (index) -> !fir.shape<1>
+ %box = fir.embox %alloc(%shape) : (!fir.heap<!fir.array<?xf32>>, !fir.shape<1>) -> !fir.box<!fir.ptr<!fir.array<?xf32>>>
+ %desc = fir.alloca !fir.box<!fir.ptr<!fir.array<?xf32>>>
+ fir.store %box to %desc : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>
+ acc.yield %desc, %alloc : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>, !fir.heap<!fir.array<?xf32>>
+ } copy {
+ ^bb0(%arg0: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>,
+ %arg1: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>):
+ acc.terminator
+ } destroy {
+ ^bb0(%arg0: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>,
+ %arg1: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>,
+ %owned: !fir.heap<!fir.array<?xf32>>):
+ fir.freemem %owned : !fir.heap<!fir.array<?xf32>>
+ acc.terminator
+ }
+
+ func.func @firstprivate_pointer(
+ %arg0: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>,
+ %arg1: !fir.box<!fir.ptr<!fir.array<?xf32>>>) {
+ %priv = acc.firstprivate varPtr(%arg0 : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>) recipe(@firstprivatization_pointer) name("p") -> !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>
+ acc.parallel firstprivate(%priv : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>) {
+ fir.store %arg1 to %priv : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>
+ acc.yield
+ }
+ return
+ }
}
// Verify that the firstprivate was materialized into a copy outside the kernel
@@ -54,3 +86,14 @@ module {
// CHECK-MAP: fir.alloca i32 {{.*}}acc.var_name = #acc.var_name<"t">
// CHECK-MAP: fir.store {{.*}} to {{.*}} : !fir.ref<i32>
// CHECK-MAP: arith.addi {{.*}} %c1
+
+// CHECK-LABEL: func.func @firstprivate_pointer
+// CHECK: %[[ALLOC:.*]] = fir.allocmem !fir.array<?xf32>
+// CHECK: %[[DESC:.*]] = fir.alloca !fir.box<!fir.ptr<!fir.array<?xf32>>>
+// CHECK: fir.store %arg1 to %[[DESC]]
+// CHECK-NOT: fir.box_addr
+// CHECK: fir.freemem %[[ALLOC]] : !fir.heap<!fir.array<?xf32>>
+
+// CHECK-MAP-LABEL: func.func @firstprivate_pointer
+// CHECK-MAP: %[[ALLOC:.*]] = fir.allocmem !fir.array<?xf32>
+// CHECK-MAP: fir.freemem %[[ALLOC]] : !fir.heap<!fir.array<?xf32>>
diff --git a/flang/test/Transforms/OpenACC/acc-recipe-materialization-private.fir b/flang/test/Transforms/OpenACC/acc-recipe-materialization-private.fir
index 97c8236472f74..10a97f080ec97 100644
--- a/flang/test/Transforms/OpenACC/acc-recipe-materialization-private.fir
+++ b/flang/test/Transforms/OpenACC/acc-recipe-materialization-private.fir
@@ -45,3 +45,38 @@ func.func @par_private_i64(%arg0 : !fir.ref<i64>) {
}
return
}
+
+acc.private.recipe @privatization_pointer : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>> init {
+^bb0(%arg0: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>):
+ %c4 = arith.constant 4 : index
+ %alloc = fir.allocmem !fir.array<?xf32>, %c4
+ %shape = fir.shape %c4 : (index) -> !fir.shape<1>
+ %box = fir.embox %alloc(%shape) : (!fir.heap<!fir.array<?xf32>>, !fir.shape<1>) -> !fir.box<!fir.ptr<!fir.array<?xf32>>>
+ %desc = fir.alloca !fir.box<!fir.ptr<!fir.array<?xf32>>>
+ fir.store %box to %desc : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>
+ acc.yield %desc, %alloc : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>, !fir.heap<!fir.array<?xf32>>
+} destroy {
+^bb0(%arg0: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>,
+ %arg1: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>,
+ %owned: !fir.heap<!fir.array<?xf32>>):
+ fir.freemem %owned : !fir.heap<!fir.array<?xf32>>
+ acc.terminator
+}
+
+// CHECK-LABEL: func.func @private_pointer
+// CHECK: %[[ALLOC:.*]] = fir.allocmem !fir.array<?xf32>
+// CHECK: %[[DESC:.*]] = fir.alloca !fir.box<!fir.ptr<!fir.array<?xf32>>>
+// CHECK: fir.store %arg1 to %[[DESC]]
+// CHECK-NOT: fir.box_addr
+// CHECK: fir.freemem %[[ALLOC]] : !fir.heap<!fir.array<?xf32>>
+
+func.func @private_pointer(
+ %arg0: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>,
+ %arg1: !fir.box<!fir.ptr<!fir.array<?xf32>>>) {
+ %priv = acc.private varPtr(%arg0 : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>) recipe(@privatization_pointer) name("p") -> !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>
+ acc.parallel private(%priv : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>) {
+ fir.store %arg1 to %priv : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xf32>>>>
+ acc.yield
+ }
+ return
+}
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
index aab852d735042..ecc0e3c3ad79b 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
@@ -1404,10 +1404,12 @@ def OpenACC_PrivateRecipeOp
default initialization. The region has an argument that contains the
original value that needs to be privatized, followed by bounds arguments
(if any) in order from innermost to outermost dimension. The region
- must yield the privatized copy.
+ must yield the privatized copy first and may yield additional values
+ that are used only for destruction.
2. The destroy region specifies how to destruct the value when it reaches
its end of life. It takes the original value, the privatized value, and
- bounds arguments (if any) in the same order as the init region.
+ any additional destruction values yielded by the init region, followed
+ by bounds arguments (if any) in the same order as the init region.
A single privatization recipe can be used for multiple operand if they have
the same type and do not require a specific default initialization.
@@ -1514,13 +1516,15 @@ def OpenACC_FirstprivateRecipeOp
default initialization. The region has an argument that contains the
original value that needs to be privatized, followed by bounds arguments
(if any) in order from innermost to outermost dimension. The region must
- yield the privatized copy.
+ yield the privatized copy first and may yield additional values that
+ are used only for destruction.
2. The copy region specifies how to copy the initial value to the newly
created private value. It takes the original value, the privatized
value, followed by bounds arguments (if any) in the same order.
3. The destroy region specifies how to destruct the value when it reaches
- its end of life. It takes the original value, the privatized value, and
- bounds arguments (if any) in the same order. It is optional.
+ its end of life. It takes the original value, the privatized value, any
+ additional destruction values yielded by the init region, and bounds
+ arguments (if any) in the same order. It is optional.
A single privatization recipe can be used for multiple operand if they have
the same type and do not require a specific default initialization.
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCTypeInterfaces.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCTypeInterfaces.td
index bbc145777b295..3b73a2e7eec17 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCTypeInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCTypeInterfaces.td
@@ -436,9 +436,10 @@ def OpenACC_MappableTypeInterface : TypeInterface<"MappableType"> {
that destruction code must be generated after the returned private
variable usages, typically in the destroy region of recipe operations
(for example, when heap allocations or temporaries requiring cleanup
- are created during initialization). When `needsDestroy` is set, callers
- should invoke `generatePrivateDestroy` in the recipe's destroy region
- with the privatized value returned by this method.
+ are created during initialization). `destroyValues` may be populated
+ with additional values that must be preserved from initialization and
+ passed to `generatePrivateDestroy`. This is useful when cleanup cannot
+ be derived from the potentially mutable privatized value.
Implementations that want certain ops to carry the correct variable name
post materialization should attach `getVarNamePlaceholder()` as the
@@ -458,7 +459,8 @@ def OpenACC_MappableTypeInterface : TypeInterface<"MappableType"> {
"::mlir::ValueRange":$extents,
"::mlir::Value":$initVal,
"::mlir::acc::VariableInfoAttr":$varInfo,
- "bool &":$needsDestroy),
+ "bool &":$needsDestroy,
+ "::llvm::SmallVectorImpl<::mlir::Value> &":$destroyValues),
/*methodBody=*/"",
/*defaultImplementation=*/[{
return {};
@@ -509,8 +511,10 @@ def OpenACC_MappableTypeInterface : TypeInterface<"MappableType"> {
produced by `generatePrivateInit`. This is typically inserted in a
recipe's destroy region, after all uses of the privatized value.
- The `privatized` value is the SSA value yielded by the init region
- (and passed as the privatized argument to the destroy region).
+ The `privatized` value is the primary SSA value yielded by the init
+ region (and passed as the privatized argument to the destroy region).
+ `destroyValues` contains any additional values yielded by init
+ specifically for destruction.
Implementations should free heap-allocated storage or perform any
cleanup required for the given type. If no destruction is required,
this function should be a no-op and return `true`.
@@ -530,6 +534,7 @@ def OpenACC_MappableTypeInterface : TypeInterface<"MappableType"> {
/*args=*/(ins "::mlir::OpBuilder &":$builder,
"::mlir::Location":$loc,
"::mlir::Value":$privatized,
+ "::mlir::ValueRange":$destroyValues,
"::mlir::ValueRange":$extents,
"::mlir::acc::VariableInfoAttr":$varInfo),
/*methodBody=*/"",
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsLoop.h b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsLoop.h
index a08d11a7ec8ab..aa37cde7c0500 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsLoop.h
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsLoop.h
@@ -23,9 +23,10 @@ namespace acc {
/// Clone an ACC region into a destination block at the given insertion point.
/// Requires a single-block source region. Maps block arguments and optional
/// result replacement: values in resultsToReplace are replaced with the
-/// operands of the cloned region's acc.yield (1:1). Erases acc.yield/terminator
-/// and merges blocks. Returns (replacement values, insertion point after
-/// clone).
+/// leading operands of the cloned region's acc.yield (1:1). Erases
+/// acc.yield/terminator and merges blocks. Returns all yielded values,
+/// including any values not used as replacements, and the insertion point after
+/// the clone.
std::pair<llvm::SmallVector<Value>, Block::iterator>
cloneACCRegionInto(Region *src, Block *dest, Block::iterator inlinePoint,
IRMapping &mapping, ValueRange resultsToReplace);
diff --git a/mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp b/mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp
index 380ddd061bc06..037aeb2cbae86 100644
--- a/mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp
+++ b/mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp
@@ -1682,7 +1682,8 @@ static LogicalResult createInitRegion(OpBuilder &builder, Location loc,
Region &initRegion, Value hostVar,
StringRef varName, ValueRange bounds,
bool &needsFree,
- acc::VariableInfoAttr &varInfo) {
+ acc::VariableInfoAttr &varInfo,
+ SmallVectorImpl<Value> &destroyValues) {
Type varType = hostVar.getType();
// Create init block with arguments: original value + bounds
@@ -1708,8 +1709,9 @@ static LogicalResult createInitRegion(OpBuilder &builder, Location loc,
auto typedVar = cast<TypedValue<MappableType>>(blockArgVar);
auto typedHostVar = cast<TypedValue<MappableType>>(hostVar);
varInfo = mappableTy.genPrivateVariableInfo(typedHostVar);
- privatizedValue = mappableTy.generatePrivateInit(
- builder, loc, typedVar, varName, bounds, {}, varInfo, needsFree);
+ privatizedValue =
+ mappableTy.generatePrivateInit(builder, loc, typedVar, varName, bounds,
+ {}, varInfo, needsFree, destroyValues);
if (!privatizedValue)
return failure();
} else {
@@ -1723,7 +1725,9 @@ static LogicalResult createInitRegion(OpBuilder &builder, Location loc,
}
// Add yield operation to init block
- acc::YieldOp::create(builder, loc, privatizedValue);
+ SmallVector<Value> initResults{privatizedValue};
+ initResults.append(destroyValues);
+ acc::YieldOp::create(builder, loc, initResults);
return success();
}
@@ -1779,14 +1783,18 @@ static LogicalResult createCopyRegion(OpBuilder &builder, Location loc,
/// Returns success if the region is populated, failure otherwise.
/// The `varInfo` carries language-specific metadata produced by
/// `createInitRegion`.
-static LogicalResult createDestroyRegion(OpBuilder &builder, Location loc,
- Region &destroyRegion, Type varType,
- Value allocRes, ValueRange bounds,
- acc::VariableInfoAttr varInfo) {
+static LogicalResult
+createDestroyRegion(OpBuilder &builder, Location loc, Region &destroyRegion,
+ Type varType, Value allocRes, ValueRange destroyValues,
+ ValueRange bounds, acc::VariableInfoAttr varInfo) {
// Create destroy block with arguments: original value + privatized value +
- // bounds
+ // values preserved for destruction + bounds.
SmallVector<Type> destroyArgTypes{varType, varType};
SmallVector<Location> destroyArgLocs{loc, loc};
+ for (Value destroyValue : destroyValues) {
+ destroyArgTypes.push_back(destroyValue.getType());
+ destroyArgLocs.push_back(loc);
+ }
for (Value bound : bounds) {
destroyArgTypes.push_back(bound.getType());
destroyArgLocs.push_back(loc);
@@ -1800,8 +1808,12 @@ static LogicalResult createDestroyRegion(OpBuilder &builder, Location loc,
cast<TypedValue<PointerLikeType>>(destroyBlock->getArgument(1));
if (isa<MappableType>(varType)) {
auto mappableTy = cast<MappableType>(varType);
- if (!mappableTy.generatePrivateDestroy(builder, loc, varToFree, bounds,
- varInfo))
+ ValueRange destroyArgs =
+ destroyBlock->getArguments().slice(2, destroyValues.size());
+ ValueRange destroyBounds =
+ destroyBlock->getArguments().drop_front(2 + destroyValues.size());
+ if (!mappableTy.generatePrivateDestroy(builder, loc, varToFree, destroyArgs,
+ destroyBounds, varInfo))
return failure();
} else {
assert(isa<PointerLikeType>(varType) && "Expected PointerLikeType");
@@ -1883,8 +1895,10 @@ PrivateRecipeOp::createAndPopulate(OpBuilder &builder, Location loc,
// Populate the init region
bool needsFree = false;
acc::VariableInfoAttr varInfo;
+ SmallVector<Value> destroyValues;
if (failed(createInitRegion(builder, loc, recipe.getInitRegion(), hostVar,
- varName, bounds, needsFree, varInfo))) {
+ varName, bounds, needsFree, varInfo,
+ destroyValues))) {
recipe.erase();
return std::nullopt;
}
@@ -1897,7 +1911,8 @@ PrivateRecipeOp::createAndPopulate(OpBuilder &builder, Location loc,
Value allocRes = yieldOp.getOperand(0);
if (failed(createDestroyRegion(builder, loc, recipe.getDestroyRegion(),
- varType, allocRes, bounds, varInfo))) {
+ varType, allocRes, destroyValues, bounds,
+ varInfo))) {
recipe.erase();
return std::nullopt;
}
@@ -1983,8 +1998,10 @@ FirstprivateRecipeOp::createAndPopulate(OpBuilder &builder, Location loc,
// then passed through to copy/destroy so generateCopy /
// generatePrivateDestroy receive the same metadata as generatePrivateInit.
acc::VariableInfoAttr varInfo;
+ SmallVector<Value> destroyValues;
if (failed(createInitRegion(builder, loc, recipe.getInitRegion(), hostVar,
- varName, bounds, needsFree, varInfo))) {
+ varName, bounds, needsFree, varInfo,
+ destroyValues))) {
recipe.erase();
return std::nullopt;
}
@@ -2004,7 +2021,8 @@ FirstprivateRecipeOp::createAndPopulate(OpBuilder &builder, Location loc,
Value allocRes = yieldOp.getOperand(0);
if (failed(createDestroyRegion(builder, loc, recipe.getDestroyRegion(),
- varType, allocRes, bounds, varInfo))) {
+ varType, allocRes, destroyValues, bounds,
+ varInfo))) {
recipe.erase();
return std::nullopt;
}
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
index e1775ce3706e0..d50cd1ef85751 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
@@ -127,15 +127,17 @@ static void resolveVarNamePlaceholders(Block *block, Block::iterator ip,
StringRef name) {
StringRef placeholder = acc::getVarNamePlaceholder();
for (auto it = block->begin(); it != std::next(ip); ++it) {
- auto attr = it->getDiscardableAttrOfType<acc::VarNameAttr>(
- acc::getVarNameAttrName());
- if (attr && attr.getName() == placeholder) {
+ it->walk([&](Operation *op) {
+ auto attr = op->getDiscardableAttrOfType<acc::VarNameAttr>(
+ acc::getVarNameAttrName());
+ if (!attr || attr.getName() != placeholder)
+ return;
if (name.empty())
- it->removeDiscardableAttr(acc::getVarNameAttrName());
+ op->removeDiscardableAttr(acc::getVarNameAttrName());
else
- it->setDiscardableAttr(acc::getVarNameAttrName(),
- acc::VarNameAttr::get(it->getContext(), name));
- }
+ op->setDiscardableAttr(acc::getVarNameAttrName(),
+ acc::VarNameAttr::get(op->getContext(), name));
+ });
}
}
@@ -292,7 +294,7 @@ LogicalResult ACCRecipeMaterialization::materialize(
Block *block = ®ion.front();
auto [results, ip] = acc::cloneACCRegionInto(
&initRegion, block, block->begin(), mapping, {accPtr});
- assert(results.size() == 1 && "expected single result from init region");
+ assert(!results.empty() && "expected a result from init region");
saveVarName(op.getAccVar(), results[0]);
resolveVarNamePlaceholders(block, ip, acc::getVariableName(op.getAccVar()));
// Clone the destroy region for a private, if it exists.
@@ -306,23 +308,26 @@ LogicalResult ACCRecipeMaterialization::materialize(
Block *block = ®ion.front();
auto [results, ip] = acc::cloneACCRegionInto(
&initRegion, block, block->begin(), mapping, {accPtr});
- assert(results.size() == 1 && "expected single result from init region");
+ assert(!results.empty() && "expected a result from init region");
saveVarName(op.getAccVar(), results[0]);
resolveVarNamePlaceholders(block, ip, acc::getVariableName(op.getAccVar()));
- // We want the copy to store the origPtr to private
- results.insert(results.begin(), origPtr);
- results.append(triples);
+ // The copy only consumes the original and user-visible private value.
+ SmallVector<Value> copyArgs{origPtr, results.front()};
+ copyArgs.append(triples);
+ // Destruction also consumes any cleanup values yielded by init.
+ SmallVector<Value> destroyArgs{origPtr};
+ destroyArgs.append(results);
+ destroyArgs.append(triples);
// Clone the copy region for a firstprivate
mapping.clear();
- mapping.map(recipe.getCopyRegion().front().getArguments(), results);
+ mapping.map(recipe.getCopyRegion().front().getArguments(), copyArgs);
// Clone the copy region for a firstprivate.
Region ©Region = recipe.getCopyRegion();
setLocation(copyRegion, loc);
acc::cloneACCRegionInto(©Region, block, std::next(ip), mapping, {});
if (!recipe.getDestroyRegion().empty()) {
- // origPtr was already pushed.
- cloneDestroy(loc, recipe, block, std::prev(block->end()), results);
+ cloneDestroy(loc, recipe, block, std::prev(block->end()), destroyArgs);
}
} else if constexpr (std::is_same_v<OpTy, acc::ReductionOp>) {
auto cloneRegionIntoAccRegion = [&](Region *src, Region *dest,
diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsLoop.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsLoop.cpp
index 5d9d362f338b3..4d2d401eb3b10 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsLoop.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsLoop.cpp
@@ -130,8 +130,9 @@ cloneACCRegionInto(Region *src, Block *dest, Block::iterator inlinePoint,
for (auto [replacement, orig] :
llvm::zip(yieldOp.getOperands(), resultsToReplace)) {
replaceAllUsesInRegionWith(orig, replacement, *dest->getParent());
- replacements.push_back(replacement);
}
+ replacements.append(yieldOp.getOperands().begin(),
+ yieldOp.getOperands().end());
ip = std::prev(yieldOp->getIterator());
yieldOp.erase();
} else {
diff --git a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsLoopTest.cpp b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsLoopTest.cpp
index c7337f4967d4c..09a3cac47ce5a 100644
--- a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsLoopTest.cpp
+++ b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsLoopTest.cpp
@@ -1006,9 +1006,11 @@ TEST_F(OpenACCUtilsLoopTest, CloneACCRegionIntoWithResultReplacement) {
b.setInsertionPoint(loopBody->getTerminator());
Value replacementVal =
arith::ConstantOp::create(b, loc, b.getI32IntegerAttr(1)).getResult();
+ Value cleanupVal =
+ arith::ConstantOp::create(b, loc, b.getI32IntegerAttr(2)).getResult();
loopBody->getTerminator()->erase();
b.setInsertionPointToEnd(loopBody);
- acc::YieldOp::create(b, loc, ValueRange{replacementVal});
+ acc::YieldOp::create(b, loc, ValueRange{replacementVal, cleanupVal});
b.setInsertionPointToEnd(entry);
Value c1value =
@@ -1024,7 +1026,9 @@ TEST_F(OpenACCUtilsLoopTest, CloneACCRegionIntoWithResultReplacement) {
auto [replacements, ip] = acc::cloneACCRegionInto(
&loopOp.getRegion(), entry, entry->begin(), mapping, ValueRange{origVal});
- ASSERT_EQ(replacements.size(), 1u);
+ ASSERT_EQ(replacements.size(), 2u);
+ EXPECT_EQ(replacements[1].getDefiningOp<arith::ConstantOp>().getValue(),
+ b.getI32IntegerAttr(2));
// The addi should now use the replacement (constant 1), not origVal
bool addiUsesReplacement = false;
for (Operation &op : entry->getOperations()) {
More information about the Mlir-commits
mailing list