[flang-commits] [mlir] [flang] [acc] Initial implementation of MemoryEffects on `acc` operations (PR #75970)
Razvan Lupusoru via flang-commits
flang-commits at lists.llvm.org
Tue Dec 19 14:20:25 PST 2023
https://github.com/razvanlupusoru updated https://github.com/llvm/llvm-project/pull/75970
>From af441dae944296997fd7ad1c2f00378f3c885f4a Mon Sep 17 00:00:00 2001
From: Razvan Lupusoru <rlupusoru at nvidia.com>
Date: Thu, 14 Dec 2023 16:05:40 -0800
Subject: [PATCH 1/2] [acc] Initial implementation of MemoryEffects on `acc`
operations
The `acc` dialect operations now implement MemoryEffects interfaces
in the following ways:
- Data entry operations which may read host memory via `varPtr` are now
marked as so. The majority of them do NOT actually read the host memory.
For example, `acc.present` works on the basis of presence of pointer
and not necessarily what the data points to - so they are not marked
as reading the host memory. They still use `varPtr` though but this
dependency is reflected through ssa.
- Data clause operations which may mutate the data pointed to by
`accPtr` are marked as doing so.
- Data clause operations which update required structured or dynamic
runtime counters are marked as reading and writing the newly defined
`RuntimeCounters` resource. Some operations, like `acc.getdeviceptr`
do not actually use the runtime counters - but are marked as reading
them since the address obtained depends on the mapping operations
which do update the runtime counters. Namely, `acc.getdeviceptr` cannot
be moved across other mapping operations.
- Constructs are marked as writing to the `ConstructResource`. This
may be too strict but is needed for the following reasons: 1) Structured
constructs may not use `accPtr` and instead use `varPtr` - when this
is the case, data actions may be removed even when used. 2) Unstructured
constructs are currently used to aggregate multiple data actions. We do
not want such constructs removed or moved for now.
- Terminators are marked as `Pure` as in other dialects.
The current approach has the following limitations which may require
further improvements:
- Subsequent `acc.copyin` operations on same data do not actually read
host memory pointed to by `varPtr` but are still marked as so.
- Two `acc.delete` operations on same data may not mutate `accPtr` until
the runtime counters are zero (but are still marked as mutating).
- The `varPtrPtr` argument, when present, points to the address of
location of `varPtr`. When mapping to target device, an `accPtrPtr`
needs computed and this memory is mutated. This effect is not captured
since the current operations do not produce `accPtrPtr`.
- Runtime counter effects are imprecise since two operations with
differing `varPtr` increment/decrement different counters. Additionally,
operations with `varPtrPtr` mutate attachment counters.
- The `ConstructResource` is too strict and likely can be relaxed with
better modeling.
---
flang/lib/Lower/OpenACC.cpp | 50 +++--
mlir/include/mlir/Dialect/OpenACC/OpenACC.h | 54 ++++-
.../mlir/Dialect/OpenACC/OpenACCOps.td | 212 +++++++++++++-----
mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp | 75 ++++++-
4 files changed, 309 insertions(+), 82 deletions(-)
diff --git a/flang/lib/Lower/OpenACC.cpp b/flang/lib/Lower/OpenACC.cpp
index 75432db33a790d..fae54eefb02f70 100644
--- a/flang/lib/Lower/OpenACC.cpp
+++ b/flang/lib/Lower/OpenACC.cpp
@@ -218,14 +218,18 @@ static void createDeclareDeallocFuncWithArg(
builder.create<mlir::acc::DeclareExitOp>(
loc, mlir::Value{}, mlir::ValueRange(entryOp.getAccPtr()));
- mlir::Value varPtr;
if constexpr (std::is_same_v<ExitOp, mlir::acc::CopyoutOp> ||
std::is_same_v<ExitOp, mlir::acc::UpdateHostOp>)
- varPtr = entryOp.getVarPtr();
- builder.create<ExitOp>(entryOp.getLoc(), entryOp.getAccPtr(), varPtr,
- entryOp.getBounds(), entryOp.getDataClause(),
- /*structured=*/false, /*implicit=*/false,
- builder.getStringAttr(*entryOp.getName()));
+ builder.create<ExitOp>(entryOp.getLoc(), entryOp.getAccPtr(),
+ entryOp.getVarPtr(), entryOp.getBounds(),
+ entryOp.getDataClause(),
+ /*structured=*/false, /*implicit=*/false,
+ builder.getStringAttr(*entryOp.getName()));
+ else
+ builder.create<ExitOp>(entryOp.getLoc(), entryOp.getAccPtr(),
+ entryOp.getBounds(), entryOp.getDataClause(),
+ /*structured=*/false, /*implicit=*/false,
+ builder.getStringAttr(*entryOp.getName()));
// Generate the post dealloc function.
modBuilder.setInsertionPointAfter(preDeallocOp);
@@ -368,14 +372,17 @@ static void genDataExitOperations(fir::FirOpBuilder &builder,
for (mlir::Value operand : operands) {
auto entryOp = mlir::dyn_cast_or_null<EntryOp>(operand.getDefiningOp());
assert(entryOp && "data entry op expected");
- mlir::Value varPtr;
if constexpr (std::is_same_v<ExitOp, mlir::acc::CopyoutOp> ||
std::is_same_v<ExitOp, mlir::acc::UpdateHostOp>)
- varPtr = entryOp.getVarPtr();
- builder.create<ExitOp>(entryOp.getLoc(), entryOp.getAccPtr(), varPtr,
- entryOp.getBounds(), entryOp.getDataClause(),
- structured, entryOp.getImplicit(),
- builder.getStringAttr(*entryOp.getName()));
+ builder.create<ExitOp>(
+ entryOp.getLoc(), entryOp.getAccPtr(), entryOp.getVarPtr(),
+ entryOp.getBounds(), entryOp.getDataClause(), structured,
+ entryOp.getImplicit(), builder.getStringAttr(*entryOp.getName()));
+ else
+ builder.create<ExitOp>(entryOp.getLoc(), entryOp.getAccPtr(),
+ entryOp.getBounds(), entryOp.getDataClause(),
+ structured, entryOp.getImplicit(),
+ builder.getStringAttr(*entryOp.getName()));
}
}
@@ -2840,9 +2847,8 @@ static void createDeclareGlobalOp(mlir::OpBuilder &modBuilder,
else
builder.create<DeclareOp>(loc, mlir::Value{},
mlir::ValueRange(entryOp.getAccPtr()));
- mlir::Value varPtr;
if constexpr (std::is_same_v<GlobalOp, mlir::acc::GlobalDestructorOp>) {
- builder.create<ExitOp>(entryOp.getLoc(), entryOp.getAccPtr(), varPtr,
+ builder.create<ExitOp>(entryOp.getLoc(), entryOp.getAccPtr(),
entryOp.getBounds(), entryOp.getDataClause(),
/*structured=*/false, /*implicit=*/false,
builder.getStringAttr(*entryOp.getName()));
@@ -2930,14 +2936,18 @@ static void createDeclareDeallocFunc(mlir::OpBuilder &modBuilder,
builder.create<mlir::acc::DeclareExitOp>(
loc, mlir::Value{}, mlir::ValueRange(entryOp.getAccPtr()));
- mlir::Value varPtr;
if constexpr (std::is_same_v<ExitOp, mlir::acc::CopyoutOp> ||
std::is_same_v<ExitOp, mlir::acc::UpdateHostOp>)
- varPtr = entryOp.getVarPtr();
- builder.create<ExitOp>(entryOp.getLoc(), entryOp.getAccPtr(), varPtr,
- entryOp.getBounds(), entryOp.getDataClause(),
- /*structured=*/false, /*implicit=*/false,
- builder.getStringAttr(*entryOp.getName()));
+ builder.create<ExitOp>(entryOp.getLoc(), entryOp.getAccPtr(),
+ entryOp.getVarPtr(), entryOp.getBounds(),
+ entryOp.getDataClause(),
+ /*structured=*/false, /*implicit=*/false,
+ builder.getStringAttr(*entryOp.getName()));
+ else
+ builder.create<ExitOp>(entryOp.getLoc(), entryOp.getAccPtr(),
+ entryOp.getBounds(), entryOp.getDataClause(),
+ /*structured=*/false, /*implicit=*/false,
+ builder.getStringAttr(*entryOp.getName()));
// Generate the post dealloc function.
modBuilder.setInsertionPointAfter(preDeallocOp);
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACC.h b/mlir/include/mlir/Dialect/OpenACC/OpenACC.h
index 4dc94782c1c9b5..36daf8de235f34 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACC.h
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACC.h
@@ -46,14 +46,22 @@
mlir::acc::UseDeviceOp, mlir::acc::ReductionOp, \
mlir::acc::DeclareDeviceResidentOp, mlir::acc::DeclareLinkOp, \
mlir::acc::CacheOp
+#define ACC_DATA_EXIT_OPS \
+ mlir::acc::CopyoutOp, mlir::acc::DeleteOp, mlir::acc::DetachOp, \
+ mlir::acc::UpdateHostOp
+#define ACC_DATA_CLAUSE_OPS ACC_DATA_ENTRY_OPS, ACC_DATA_EXIT_OPS
#define ACC_COMPUTE_CONSTRUCT_OPS \
mlir::acc::ParallelOp, mlir::acc::KernelsOp, mlir::acc::SerialOp
#define ACC_COMPUTE_CONSTRUCT_AND_LOOP_OPS \
ACC_COMPUTE_CONSTRUCT_OPS, mlir::acc::LoopOp
+#define OPENACC_DATA_CONSTRUCT_STRUCTURED_OPS \
+ mlir::acc::DataOp, mlir::acc::DeclareOp
+#define ACC_DATA_CONSTRUCT_UNSTRUCTURED_OPS \
+ mlir::acc::EnterDataOp, mlir::acc::ExitDataOp, mlir::acc::UpdateOp, \
+ mlir::acc::HostDataOp, mlir::acc::DeclareEnterOp, \
+ mlir::acc::DeclareExitOp
#define ACC_DATA_CONSTRUCT_OPS \
- mlir::acc::DataOp, mlir::acc::EnterDataOp, mlir::acc::ExitDataOp, \
- mlir::acc::UpdateOp, mlir::acc::HostDataOp, mlir::acc::DeclareEnterOp, \
- mlir::acc::DeclareExitOp, mlir::acc::DeclareOp
+ OPENACC_DATA_CONSTRUCT_STRUCTURED_OPS, ACC_DATA_CONSTRUCT_UNSTRUCTURED_OPS
#define ACC_COMPUTE_AND_DATA_CONSTRUCT_OPS \
ACC_COMPUTE_CONSTRUCT_OPS, ACC_DATA_CONSTRUCT_OPS
#define ACC_COMPUTE_LOOP_AND_DATA_CONSTRUCT_OPS \
@@ -73,9 +81,27 @@ namespace acc {
/// combined and the final mapping value would be 5 (4 | 1).
enum OpenACCExecMapping { NONE = 0, VECTOR = 1, WORKER = 2, GANG = 4 };
-/// Used to obtain the `varPtr` from a data entry operation.
-/// Returns empty value if not a data entry operation.
-mlir::Value getVarPtr(mlir::Operation *accDataEntryOp);
+/// Used to obtain the `varPtr` from a data clause operation.
+/// Returns empty value if not a data clause operation or is a data exit
+/// operation with no `varPtr`.
+mlir::Value getVarPtr(mlir::Operation *accDataClauseOp);
+
+/// Used to obtain the `accPtr` from a data clause operation.
+/// When a data entry operation, it obtains its result `accPtr` value.
+/// If a data exit operation, it obtains its operand `accPtr` value.
+/// Returns empty value if not a data clause operation.
+mlir::Value getAccPtr(mlir::Operation *accDataClauseOp);
+
+/// Used to obtain the `varPtrPtr` from a data clause operation.
+/// Returns empty value if not a data clause operation.
+mlir::Value getVarPtrPtr(mlir::Operation *accDataClauseOp);
+
+/// Used to obtain `bounds` from an acc data clause operation.
+/// Returns an empty vector if there are no bounds.
+mlir::SmallVector<mlir::Value> getBounds(mlir::Operation *accDataClauseOp);
+
+/// Used to obtain the `name` from an acc operation.
+std::optional<llvm::StringRef> getVarName(mlir::Operation *accOp);
/// Used to obtain the `dataClause` from a data entry operation.
/// Returns empty optional if not a data entry operation.
@@ -87,6 +113,12 @@ getDataClause(mlir::Operation *accDataEntryOp);
/// implicit flag.
bool getImplicitFlag(mlir::Operation *accDataEntryOp);
+/// Used to get an immutable range iterating over the data operands.
+mlir::ValueRange getDataOperands(mlir::Operation *accOp);
+
+/// Used to get a mutable range iterating over the data operands.
+mlir::MutableOperandRange getMutableDataOperands(mlir::Operation *accOp);
+
/// Used to obtain the attribute name for declare.
static constexpr StringLiteral getDeclareAttrName() {
return StringLiteral("acc.declare");
@@ -100,6 +132,16 @@ static constexpr StringLiteral getRoutineInfoAttrName() {
return StringLiteral("acc.routine_info");
}
+struct RuntimeCounters
+ : public mlir::SideEffects::Resource::Base<RuntimeCounters> {
+ mlir::StringRef getName() final { return "AccRuntimeCounters"; }
+};
+
+struct ConstructResource
+ : public mlir::SideEffects::Resource::Base<ConstructResource> {
+ mlir::StringRef getName() final { return "AccConstructResource"; }
+};
+
} // namespace acc
} // namespace mlir
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
index 9d48b1f1c3f9af..4abc0d62dbfdae 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
@@ -179,6 +179,15 @@ def OpenACC_DeviceTypeAttr : EnumAttr<OpenACC_Dialect,
let assemblyFormat = [{ ```<` $value `>` }];
}
+// Define a resource for the OpenACC runtime counters.
+def OpenACC_RuntimeCounters : Resource<"::mlir::acc::RuntimeCounters">;
+
+// Define a resource for the OpenACC constructs.
+// Useful to ensure that the constructs are not removed (even though
+// the data semantics are encoded in the operations linked via their
+// `dataOperands` list).
+def OpenACC_ConstructResource : Resource<"::mlir::acc::ConstructResource">;
+
// Used for data specification in data clauses (2.7.1).
// Either (or both) extent and upperbound must be specified.
def OpenACC_DataBoundsOp : OpenACC_Op<"bounds",
@@ -250,18 +259,18 @@ def OpenACC_DataBoundsOp : OpenACC_Op<"bounds",
//
// The bounds are represented in rank order. Rank 0 (inner-most dimension) is
// the first.
+//
class OpenACC_DataEntryOp<string mnemonic, string clause, string extraDescription,
- list<Trait> traits = []> :
+ list<Trait> traits = [], dag additionalArgs = (ins)> :
OpenACC_Op<mnemonic, !listconcat(traits,
[AttrSizedOperandSegments])> {
- let arguments = (ins OpenACC_PointerLikeTypeInterface:$varPtr,
- Optional<OpenACC_PointerLikeTypeInterface>:$varPtrPtr,
+ let arguments = !con(additionalArgs,
+ (ins Optional<OpenACC_PointerLikeTypeInterface>:$varPtrPtr,
Variadic<OpenACC_DataBoundsType>:$bounds, /* rank-0 to rank-{n-1} */
DefaultValuedAttr<OpenACC_DataClauseAttr,clause>:$dataClause,
DefaultValuedAttr<BoolAttr, "true">:$structured,
DefaultValuedAttr<BoolAttr, "false">:$implicit,
- OptionalAttr<StrAttr>:$name);
- let results = (outs OpenACC_PointerLikeTypeInterface:$accPtr);
+ OptionalAttr<StrAttr>:$name));
let description = !strconcat(extraDescription, [{
Description of arguments:
@@ -299,50 +308,73 @@ class OpenACC_DataEntryOp<string mnemonic, string clause, string extraDescriptio
// 2.5.13 private clause
//===----------------------------------------------------------------------===//
def OpenACC_PrivateOp : OpenACC_DataEntryOp<"private",
- "mlir::acc::DataClause::acc_private", ""> {
+ "mlir::acc::DataClause::acc_private", "", [],
+ (ins OpenACC_PointerLikeTypeInterface:$varPtr)> {
let summary = "Represents private semantics for acc private clause.";
+ let results = (outs Arg<OpenACC_PointerLikeTypeInterface,
+ "Address of device variable",[MemWrite]>:$accPtr);
}
//===----------------------------------------------------------------------===//
// 2.5.14 firstprivate clause
//===----------------------------------------------------------------------===//
def OpenACC_FirstprivateOp : OpenACC_DataEntryOp<"firstprivate",
- "mlir::acc::DataClause::acc_firstprivate", ""> {
+ "mlir::acc::DataClause::acc_firstprivate", "", [],
+ (ins Arg<OpenACC_PointerLikeTypeInterface,"Address of variable",[MemRead]>:$varPtr)> {
let summary = "Represents firstprivate semantic for the acc firstprivate "
"clause.";
+ let results = (outs Arg<OpenACC_PointerLikeTypeInterface,
+ "Address of device variable",[MemWrite]>:$accPtr);
}
//===----------------------------------------------------------------------===//
// 2.5.15 reduction clause
//===----------------------------------------------------------------------===//
def OpenACC_ReductionOp : OpenACC_DataEntryOp<"reduction",
- "mlir::acc::DataClause::acc_reduction", ""> {
+ "mlir::acc::DataClause::acc_reduction", "",
+ [MemoryEffects<[MemRead<OpenACC_RuntimeCounters>,
+ MemWrite<OpenACC_RuntimeCounters>]>],
+ (ins Arg<OpenACC_PointerLikeTypeInterface,"Address of variable",[MemRead]>:$varPtr)> {
let summary = "Represents reduction semantics for acc reduction clause.";
+ let results = (outs Arg<OpenACC_PointerLikeTypeInterface,
+ "Address of device variable",[MemWrite]>:$accPtr);
}
//===----------------------------------------------------------------------===//
// 2.7.4 deviceptr clause
//===----------------------------------------------------------------------===//
def OpenACC_DevicePtrOp : OpenACC_DataEntryOp<"deviceptr",
- "mlir::acc::DataClause::acc_deviceptr", ""> {
+ "mlir::acc::DataClause::acc_deviceptr", "",
+ [MemoryEffects<[MemRead<OpenACC_RuntimeCounters>]>],
+ (ins OpenACC_PointerLikeTypeInterface:$varPtr)> {
let summary = "Specifies that the variable pointer is a device pointer.";
+ let results = (outs OpenACC_PointerLikeTypeInterface:$accPtr);
}
//===----------------------------------------------------------------------===//
// 2.7.5 present clause
//===----------------------------------------------------------------------===//
def OpenACC_PresentOp : OpenACC_DataEntryOp<"present",
- "mlir::acc::DataClause::acc_present", ""> {
+ "mlir::acc::DataClause::acc_present", "",
+ [MemoryEffects<[MemRead<OpenACC_RuntimeCounters>,
+ MemWrite<OpenACC_RuntimeCounters>]>],
+ (ins OpenACC_PointerLikeTypeInterface:$varPtr)> {
let summary = "Specifies that the variable is already present on device.";
+ let results = (outs OpenACC_PointerLikeTypeInterface:$accPtr);
}
//===----------------------------------------------------------------------===//
// 2.7.7 copyin clause
//===----------------------------------------------------------------------===//
def OpenACC_CopyinOp : OpenACC_DataEntryOp<"copyin",
- "mlir::acc::DataClause::acc_copyin", ""> {
+ "mlir::acc::DataClause::acc_copyin", "",
+ [MemoryEffects<[MemRead<OpenACC_RuntimeCounters>,
+ MemWrite<OpenACC_RuntimeCounters>]>],
+ (ins Arg<OpenACC_PointerLikeTypeInterface,"Address of variable",[MemRead]>:$varPtr)> {
let summary = "Represents copyin semantics for acc data clauses like acc "
"copyin and acc copy.";
+ let results = (outs Arg<OpenACC_PointerLikeTypeInterface,
+ "Address of device variable",[MemWrite]>:$accPtr);
let extraClassDeclaration = [{
/// Check if this is a copyin with readonly modifier.
@@ -354,9 +386,14 @@ def OpenACC_CopyinOp : OpenACC_DataEntryOp<"copyin",
// 2.7.9 create clause
//===----------------------------------------------------------------------===//
def OpenACC_CreateOp : OpenACC_DataEntryOp<"create",
- "mlir::acc::DataClause::acc_create", ""> {
+ "mlir::acc::DataClause::acc_create", "",
+ [MemoryEffects<[MemRead<OpenACC_RuntimeCounters>,
+ MemWrite<OpenACC_RuntimeCounters>]>],
+ (ins OpenACC_PointerLikeTypeInterface:$varPtr)> {
let summary = "Represents create semantics for acc data clauses like acc "
"create and acc copyout.";
+ let results = (outs Arg<OpenACC_PointerLikeTypeInterface,
+ "Address of device variable",[MemWrite]>:$accPtr);
let extraClassDeclaration = [{
/// Check if this is a create with zero modifier.
@@ -368,18 +405,26 @@ def OpenACC_CreateOp : OpenACC_DataEntryOp<"create",
// 2.7.10 no_create clause
//===----------------------------------------------------------------------===//
def OpenACC_NoCreateOp : OpenACC_DataEntryOp<"nocreate",
- "mlir::acc::DataClause::acc_no_create", ""> {
+ "mlir::acc::DataClause::acc_no_create", "",
+ [MemoryEffects<[MemRead<OpenACC_RuntimeCounters>,
+ MemWrite<OpenACC_RuntimeCounters>]>],
+ (ins OpenACC_PointerLikeTypeInterface:$varPtr)> {
let summary = "Represents acc no_create semantics.";
+ let results = (outs OpenACC_PointerLikeTypeInterface:$accPtr);
}
//===----------------------------------------------------------------------===//
// 2.7.12 attach clause
//===----------------------------------------------------------------------===//
def OpenACC_AttachOp : OpenACC_DataEntryOp<"attach",
- "mlir::acc::DataClause::acc_attach", ""> {
+ "mlir::acc::DataClause::acc_attach", "",
+ [MemoryEffects<[MemRead<OpenACC_RuntimeCounters>,
+ MemWrite<OpenACC_RuntimeCounters>]>],
+ (ins Arg<OpenACC_PointerLikeTypeInterface,"Address of variable",[MemRead]>:$varPtr)> {
let summary = "Represents acc attach semantics which updates a pointer in "
"device memory with the corresponding device address of the "
"pointee.";
+ let results = (outs OpenACC_PointerLikeTypeInterface:$accPtr);
}
//===----------------------------------------------------------------------===//
@@ -397,8 +442,10 @@ def OpenACC_GetDevicePtrOp : OpenACC_DataEntryOp<"getdeviceptr",
operation is not visible. This operation can have a `dataClause` argument
that is any of the valid `mlir::acc::DataClause` entries.
\
- }]> {
+ }], [MemoryEffects<[MemRead<OpenACC_RuntimeCounters>]>],
+ (ins OpenACC_PointerLikeTypeInterface:$varPtr)> {
let summary = "Gets device address if variable exists on device.";
+ let results = (outs OpenACC_PointerLikeTypeInterface:$accPtr);
let hasVerifier = 0;
}
@@ -406,41 +453,55 @@ def OpenACC_GetDevicePtrOp : OpenACC_DataEntryOp<"getdeviceptr",
// 2.14.4 device clause
//===----------------------------------------------------------------------===//
def OpenACC_UpdateDeviceOp : OpenACC_DataEntryOp<"update_device",
- "mlir::acc::DataClause::acc_update_device", ""> {
+ "mlir::acc::DataClause::acc_update_device", "", [],
+ (ins Arg<OpenACC_PointerLikeTypeInterface,"Address of variable",[MemRead]>:$varPtr)> {
let summary = "Represents acc update device semantics.";
+ let results = (outs Arg<OpenACC_PointerLikeTypeInterface,
+ "Address of device variable",[MemWrite]>:$accPtr);
}
//===----------------------------------------------------------------------===//
// 2.8 use_device clause
//===----------------------------------------------------------------------===//
def OpenACC_UseDeviceOp : OpenACC_DataEntryOp<"use_device",
- "mlir::acc::DataClause::acc_use_device", ""> {
+ "mlir::acc::DataClause::acc_use_device", "",
+ [MemoryEffects<[MemRead<OpenACC_RuntimeCounters>]>],
+ (ins OpenACC_PointerLikeTypeInterface:$varPtr)> {
let summary = "Represents acc use_device semantics.";
+ let results = (outs OpenACC_PointerLikeTypeInterface:$accPtr);
}
//===----------------------------------------------------------------------===//
// 2.13.1 device_resident clause
//===----------------------------------------------------------------------===//
def OpenACC_DeclareDeviceResidentOp : OpenACC_DataEntryOp<"declare_device_resident",
- "mlir::acc::DataClause::acc_declare_device_resident", ""> {
+ "mlir::acc::DataClause::acc_declare_device_resident", "",
+ [MemoryEffects<[MemWrite<OpenACC_RuntimeCounters>]>],
+ (ins Arg<OpenACC_PointerLikeTypeInterface,"Address of variable",[MemRead]>:$varPtr)> {
let summary = "Represents acc declare device_resident semantics.";
+ let results = (outs OpenACC_PointerLikeTypeInterface:$accPtr);
}
//===----------------------------------------------------------------------===//
// 2.13.3 link clause
//===----------------------------------------------------------------------===//
def OpenACC_DeclareLinkOp : OpenACC_DataEntryOp<"declare_link",
- "mlir::acc::DataClause::acc_declare_link", ""> {
+ "mlir::acc::DataClause::acc_declare_link", "",
+ [MemoryEffects<[MemWrite<OpenACC_RuntimeCounters>]>],
+ (ins Arg<OpenACC_PointerLikeTypeInterface,"Address of variable",[MemRead]>:$varPtr)> {
let summary = "Represents acc declare link semantics.";
+ let results = (outs OpenACC_PointerLikeTypeInterface:$accPtr);
}
//===----------------------------------------------------------------------===//
// 2.10 cache directive
//===----------------------------------------------------------------------===//
def OpenACC_CacheOp : OpenACC_DataEntryOp<"cache",
- "mlir::acc::DataClause::acc_cache", ""> {
+ "mlir::acc::DataClause::acc_cache", "", [NoMemoryEffect],
+ (ins OpenACC_PointerLikeTypeInterface:$varPtr)> {
let summary = "Represents the cache directive that is associated with a "
"loop.";
+ let results = (outs OpenACC_PointerLikeTypeInterface:$accPtr);
let extraClassDeclaration = [{
/// Check if this is a cache with readonly modifier.
@@ -454,20 +515,17 @@ def OpenACC_CacheOp : OpenACC_DataEntryOp<"cache",
// terminology used in this dialect. It refers to data operations that will appear
// after data or compute region. It will be used as the base of acc dialect
// operations for the following OpenACC data clauses: copyout, detach, delete.
-class OpenACC_DataExitOp<string mnemonic, string clause, list<Trait> traits = []> :
- OpenACC_Op<mnemonic, !listconcat(traits,
- [AttrSizedOperandSegments])> {
- let arguments = (ins OpenACC_PointerLikeTypeInterface:$accPtr,
- Optional<OpenACC_PointerLikeTypeInterface>:$varPtr,
- Variadic<OpenACC_DataBoundsType>:$bounds,
+class OpenACC_DataExitOp<string mnemonic, string clause, string extraDescription,
+ list<Trait> traits = [], dag additionalArgs = (ins)> :
+ OpenACC_Op<mnemonic, !listconcat(traits, [])> {
+ let arguments = !con(additionalArgs,
+ (ins Variadic<OpenACC_DataBoundsType>:$bounds,
DefaultValuedAttr<OpenACC_DataClauseAttr,clause>:$dataClause,
DefaultValuedAttr<BoolAttr, "true">:$structured,
DefaultValuedAttr<BoolAttr, "false">:$implicit,
- OptionalAttr<StrAttr>:$name);
+ OptionalAttr<StrAttr>:$name));
- let description = [{
- - `varPtr`: The address of variable to copy back to. This only applies to
- `acc.copyout`
+ let description = !strconcat(extraDescription, [{
- `accPtr`: The acc address of variable. This is the link from the data-entry
operation used.
- `bounds`: Used when copying just slice of array or array's bounds are not
@@ -483,7 +541,7 @@ class OpenACC_DataExitOp<string mnemonic, string clause, list<Trait> traits = []
- `implicit`: Whether this is an implicitly generated operation, such as copies
done to satisfy "Variables with Implicitly Determined Data Attributes" in 2.6.2.
- `name`: Holds the name of variable as specified in user clause (including bounds).
- }];
+ }]);
let assemblyFormat = [{
`accPtr` `(` $accPtr `:` type($accPtr) `)`
@@ -500,36 +558,71 @@ class OpenACC_DataExitOp<string mnemonic, string clause, list<Trait> traits = []
// 2.7.8 copyout clause
//===----------------------------------------------------------------------===//
def OpenACC_CopyoutOp : OpenACC_DataExitOp<"copyout",
- "mlir::acc::DataClause::acc_copyout"> {
+ "mlir::acc::DataClause::acc_copyout",
+ "- `varPtr`: The address of variable to copy back to.",
+ [MemoryEffects<[MemRead<OpenACC_RuntimeCounters>,
+ MemWrite<OpenACC_RuntimeCounters>]>],
+ (ins Arg<OpenACC_PointerLikeTypeInterface,"Address of device variable",[MemRead]>:$accPtr,
+ Arg<OpenACC_PointerLikeTypeInterface,"Address of variable",[MemWrite]>:$varPtr)> {
let summary = "Represents acc copyout semantics - reverse of copyin.";
let extraClassDeclaration = [{
/// Check if this is a copyout with zero modifier.
bool isCopyoutZero();
}];
+
+ let assemblyFormat = [{
+ `accPtr` `(` $accPtr `:` type($accPtr) `)`
+ (`bounds` `(` $bounds^ `)` )?
+ `to` `varPtr` `(` $varPtr `:` type($varPtr) `)`
+ attr-dict
+ }];
}
//===----------------------------------------------------------------------===//
// 2.7.11 delete clause
//===----------------------------------------------------------------------===//
def OpenACC_DeleteOp : OpenACC_DataExitOp<"delete",
- "mlir::acc::DataClause::acc_delete"> {
+ "mlir::acc::DataClause::acc_delete", "",
+ [MemoryEffects<[MemRead<OpenACC_RuntimeCounters>,
+ MemWrite<OpenACC_RuntimeCounters>]>],
+ (ins Arg<OpenACC_PointerLikeTypeInterface,"Address of device variable",[MemRead]>:$accPtr)> {
let summary = "Represents acc delete semantics - reverse of create.";
+
+ let assemblyFormat = [{
+ `accPtr` `(` $accPtr `:` type($accPtr) `)`
+ (`bounds` `(` $bounds^ `)` )?
+ attr-dict
+ }];
}
//===----------------------------------------------------------------------===//
// 2.7.13 detach clause
//===----------------------------------------------------------------------===//
def OpenACC_DetachOp : OpenACC_DataExitOp<"detach",
- "mlir::acc::DataClause::acc_detach"> {
+ "mlir::acc::DataClause::acc_detach", "",
+ [MemoryEffects<[MemRead<OpenACC_RuntimeCounters>,
+ MemWrite<OpenACC_RuntimeCounters>]>],
+ (ins Arg<OpenACC_PointerLikeTypeInterface,"Address of device variable",[MemRead]>:$accPtr)> {
let summary = "Represents acc detach semantics - reverse of attach.";
+
+ let assemblyFormat = [{
+ `accPtr` `(` $accPtr `:` type($accPtr) `)`
+ (`bounds` `(` $bounds^ `)` )?
+ attr-dict
+ }];
}
//===----------------------------------------------------------------------===//
// 2.14.4 host clause
//===----------------------------------------------------------------------===//
def OpenACC_UpdateHostOp : OpenACC_DataExitOp<"update_host",
- "mlir::acc::DataClause::acc_update_host"> {
+ "mlir::acc::DataClause::acc_update_host",
+ "- `varPtr`: The address of variable to copy back to.",
+ [MemoryEffects<[MemRead<OpenACC_RuntimeCounters>,
+ MemWrite<OpenACC_RuntimeCounters>]>],
+ (ins Arg<OpenACC_PointerLikeTypeInterface,"Address of device variable",[MemRead]>:$accPtr,
+ Arg<OpenACC_PointerLikeTypeInterface,"Address of variable",[MemWrite]>:$varPtr)> {
let summary = "Represents acc update host semantics.";
let extraClassDeclaration = [{
/// Check if this is an acc update self.
@@ -537,6 +630,13 @@ def OpenACC_UpdateHostOp : OpenACC_DataExitOp<"update_host",
return getDataClause() == acc::DataClause::acc_update_self;
}
}];
+
+ let assemblyFormat = [{
+ `accPtr` `(` $accPtr `:` type($accPtr) `)`
+ (`bounds` `(` $bounds^ `)` )?
+ `to` `varPtr` `(` $varPtr `:` type($varPtr) `)`
+ attr-dict
+ }];
}
//===----------------------------------------------------------------------===//
@@ -748,7 +848,8 @@ def OpenACC_ReductionRecipeOp : OpenACC_Op<"reduction.recipe",
//===----------------------------------------------------------------------===//
def OpenACC_ParallelOp : OpenACC_Op<"parallel",
- [AttrSizedOperandSegments, RecursiveMemoryEffects]> {
+ [AttrSizedOperandSegments, RecursiveMemoryEffects,
+ MemoryEffects<[MemWrite<OpenACC_ConstructResource>]>]> {
let summary = "parallel construct";
let description = [{
The "acc.parallel" operation represents a parallel construct block. It has
@@ -824,7 +925,8 @@ def OpenACC_ParallelOp : OpenACC_Op<"parallel",
//===----------------------------------------------------------------------===//
def OpenACC_SerialOp : OpenACC_Op<"serial",
- [AttrSizedOperandSegments, RecursiveMemoryEffects]> {
+ [AttrSizedOperandSegments, RecursiveMemoryEffects,
+ MemoryEffects<[MemWrite<OpenACC_ConstructResource>]>]> {
let summary = "serial construct";
let description = [{
The "acc.serial" operation represents a serial construct block. It has
@@ -893,7 +995,8 @@ def OpenACC_SerialOp : OpenACC_Op<"serial",
//===----------------------------------------------------------------------===//
def OpenACC_KernelsOp : OpenACC_Op<"kernels",
- [AttrSizedOperandSegments, RecursiveMemoryEffects]> {
+ [AttrSizedOperandSegments, RecursiveMemoryEffects,
+ MemoryEffects<[MemWrite<OpenACC_ConstructResource>]>]> {
let summary = "kernels construct";
let description = [{
The "acc.kernels" operation represents a kernels construct block. It has
@@ -955,7 +1058,8 @@ def OpenACC_KernelsOp : OpenACC_Op<"kernels",
//===----------------------------------------------------------------------===//
def OpenACC_DataOp : OpenACC_Op<"data",
- [AttrSizedOperandSegments, RecursiveMemoryEffects]> {
+ [AttrSizedOperandSegments, RecursiveMemoryEffects,
+ MemoryEffects<[MemWrite<OpenACC_ConstructResource>]>]> {
let summary = "data construct";
let description = [{
@@ -1008,7 +1112,7 @@ def OpenACC_DataOp : OpenACC_Op<"data",
let hasVerifier = 1;
}
-def OpenACC_TerminatorOp : OpenACC_Op<"terminator", [Terminator]> {
+def OpenACC_TerminatorOp : OpenACC_Op<"terminator", [Pure, Terminator]> {
let summary = "Generic terminator for OpenACC regions";
let description = [{
@@ -1025,7 +1129,8 @@ def OpenACC_TerminatorOp : OpenACC_Op<"terminator", [Terminator]> {
// 2.6.6 Enter Data Directive
//===----------------------------------------------------------------------===//
-def OpenACC_EnterDataOp : OpenACC_Op<"enter_data", [AttrSizedOperandSegments]> {
+def OpenACC_EnterDataOp : OpenACC_Op<"enter_data",
+ [AttrSizedOperandSegments, MemoryEffects<[MemWrite<OpenACC_ConstructResource>]>]> {
let summary = "enter data operation";
let description = [{
@@ -1073,7 +1178,8 @@ def OpenACC_EnterDataOp : OpenACC_Op<"enter_data", [AttrSizedOperandSegments]> {
// 2.6.6 Exit Data Directive
//===----------------------------------------------------------------------===//
-def OpenACC_ExitDataOp : OpenACC_Op<"exit_data", [AttrSizedOperandSegments]> {
+def OpenACC_ExitDataOp : OpenACC_Op<"exit_data",
+ [AttrSizedOperandSegments, MemoryEffects<[MemWrite<OpenACC_ConstructResource>]>]> {
let summary = "exit data operation";
let description = [{
@@ -1122,7 +1228,8 @@ def OpenACC_ExitDataOp : OpenACC_Op<"exit_data", [AttrSizedOperandSegments]> {
// 2.8 Host_Data Construct
//===----------------------------------------------------------------------===//
-def OpenACC_HostDataOp : OpenACC_Op<"host_data", [AttrSizedOperandSegments]> {
+def OpenACC_HostDataOp : OpenACC_Op<"host_data",
+ [AttrSizedOperandSegments, MemoryEffects<[MemWrite<OpenACC_ConstructResource>]>]> {
let summary = "host_data construct";
let description = [{
@@ -1161,7 +1268,8 @@ def OpenACC_HostDataOp : OpenACC_Op<"host_data", [AttrSizedOperandSegments]> {
//===----------------------------------------------------------------------===//
def OpenACC_LoopOp : OpenACC_Op<"loop",
- [AttrSizedOperandSegments, RecursiveMemoryEffects]> {
+ [AttrSizedOperandSegments, RecursiveMemoryEffects,
+ MemoryEffects<[MemWrite<OpenACC_ConstructResource>]>]> {
let summary = "loop construct";
let description = [{
@@ -1244,7 +1352,7 @@ def OpenACC_LoopOp : OpenACC_Op<"loop",
}
// Yield operation for the acc.loop and acc.parallel operations.
-def OpenACC_YieldOp : OpenACC_Op<"yield", [ReturnLike, Terminator,
+def OpenACC_YieldOp : OpenACC_Op<"yield", [Pure, ReturnLike, Terminator,
ParentOneOf<["FirstprivateRecipeOp, LoopOp, ParallelOp, PrivateRecipeOp,"
"ReductionRecipeOp, SerialOp, AtomicUpdateOp"]>]> {
let summary = "Acc yield and termination operation";
@@ -1413,7 +1521,8 @@ def AtomicCaptureOp : OpenACC_Op<"atomic.capture",
// 2.13 Declare Directive
//===----------------------------------------------------------------------===//
-def OpenACC_DeclareEnterOp : OpenACC_Op<"declare_enter", []> {
+def OpenACC_DeclareEnterOp : OpenACC_Op<"declare_enter",
+ [MemoryEffects<[MemWrite<OpenACC_ConstructResource>]>]> {
let summary = "declare directive - entry to implicit data region";
let description = [{
@@ -1442,7 +1551,8 @@ def OpenACC_DeclareEnterOp : OpenACC_Op<"declare_enter", []> {
let hasVerifier = 1;
}
-def OpenACC_DeclareExitOp : OpenACC_Op<"declare_exit", [AttrSizedOperandSegments]> {
+def OpenACC_DeclareExitOp : OpenACC_Op<"declare_exit",
+ [AttrSizedOperandSegments, MemoryEffects<[MemWrite<OpenACC_ConstructResource>]>]> {
let summary = "declare directive - exit from implicit data region";
let description = [{
@@ -1543,7 +1653,8 @@ def OpenACC_GlobalDestructorOp : OpenACC_Op<"global_dtor",
let hasVerifier = 0;
}
-def OpenACC_DeclareOp : OpenACC_Op<"declare", [RecursiveMemoryEffects]> {
+def OpenACC_DeclareOp : OpenACC_Op<"declare",
+ [RecursiveMemoryEffects, MemoryEffects<[MemWrite<OpenACC_ConstructResource>]>]> {
let summary = "declare implicit region";
let description = [{
@@ -1737,11 +1848,12 @@ def OpenACC_SetOp : OpenACC_Op<"set", [AttrSizedOperandSegments]> {
// 2.14.4. Update Directive
//===----------------------------------------------------------------------===//
-def OpenACC_UpdateOp : OpenACC_Op<"update", [AttrSizedOperandSegments]> {
+def OpenACC_UpdateOp : OpenACC_Op<"update",
+ [AttrSizedOperandSegments, MemoryEffects<[MemWrite<OpenACC_ConstructResource>]>]> {
let summary = "update operation";
let description = [{
- The "acc.udpate" operation represents the OpenACC update executable
+ The `acc.update` operation represents the OpenACC update executable
directive.
As host and self clauses are synonyms, any operands for host and self are
add to $hostOperands.
diff --git a/mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp b/mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp
index 08e83cad482207..df4f7825545c2b 100644
--- a/mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp
+++ b/mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp
@@ -248,8 +248,8 @@ LogicalResult acc::DeleteOp::verify() {
return emitError(
"data clause associated with delete operation must match its intent"
" or specify original clause this operation was decomposed from");
- if (!getVarPtr() && !getAccPtr())
- return emitError("must have either host or device pointer");
+ if (!getAccPtr())
+ return emitError("must have device pointer");
return success();
}
@@ -263,8 +263,8 @@ LogicalResult acc::DetachOp::verify() {
return emitError(
"data clause associated with detach operation must match its intent"
" or specify original clause this operation was decomposed from");
- if (!getVarPtr() && !getAccPtr())
- return emitError("must have either host or device pointer");
+ if (!getAccPtr())
+ return emitError("must have device pointer");
return success();
}
@@ -1336,14 +1336,58 @@ LogicalResult acc::WaitOp::verify() {
// acc dialect utilities
//===----------------------------------------------------------------------===//
-mlir::Value mlir::acc::getVarPtr(mlir::Operation *accDataEntryOp) {
- auto varPtr{llvm::TypeSwitch<mlir::Operation *, mlir::Value>(accDataEntryOp)
+mlir::Value mlir::acc::getVarPtr(mlir::Operation *accDataClauseOp) {
+ auto varPtr{llvm::TypeSwitch<mlir::Operation *, mlir::Value>(accDataClauseOp)
.Case<ACC_DATA_ENTRY_OPS>(
[&](auto entry) { return entry.getVarPtr(); })
+ .Case<mlir::acc::CopyoutOp, mlir::acc::UpdateHostOp>(
+ [&](auto exit) { return exit.getVarPtr(); })
.Default([&](mlir::Operation *) { return mlir::Value(); })};
return varPtr;
}
+mlir::Value mlir::acc::getAccPtr(mlir::Operation *accDataClauseOp) {
+ auto accPtr{llvm::TypeSwitch<mlir::Operation *, mlir::Value>(accDataClauseOp)
+ .Case<ACC_DATA_ENTRY_OPS, ACC_DATA_EXIT_OPS>(
+ [&](auto dataClause) { return dataClause.getAccPtr(); })
+ .Default([&](mlir::Operation *) { return mlir::Value(); })};
+ return accPtr;
+}
+
+mlir::Value mlir::acc::getVarPtrPtr(mlir::Operation *accDataClauseOp) {
+ auto varPtrPtr{
+ llvm::TypeSwitch<mlir::Operation *, mlir::Value>(accDataClauseOp)
+ .Case<ACC_DATA_ENTRY_OPS>(
+ [&](auto dataClause) { return dataClause.getVarPtrPtr(); })
+ .Default([&](mlir::Operation *) { return mlir::Value(); })};
+ return varPtrPtr;
+}
+
+mlir::SmallVector<mlir::Value>
+mlir::acc::getBounds(mlir::Operation *accDataClauseOp) {
+ mlir::SmallVector<mlir::Value> bounds{
+ llvm::TypeSwitch<mlir::Operation *, mlir::SmallVector<mlir::Value>>(
+ accDataClauseOp)
+ .Case<ACC_DATA_ENTRY_OPS, ACC_DATA_EXIT_OPS>([&](auto dataClause) {
+ return mlir::SmallVector<mlir::Value>(
+ dataClause.getBounds().begin(), dataClause.getBounds().end());
+ })
+ .Default([&](mlir::Operation *) {
+ return mlir::SmallVector<mlir::Value, 0>();
+ })};
+ return bounds;
+}
+
+std::optional<llvm::StringRef> mlir::acc::getVarName(mlir::Operation *accOp) {
+ auto name{
+ llvm::TypeSwitch<mlir::Operation *, std::optional<llvm::StringRef>>(accOp)
+ .Case<ACC_DATA_ENTRY_OPS>([&](auto entry) { return entry.getName(); })
+ .Default([&](mlir::Operation *) -> std::optional<llvm::StringRef> {
+ return {};
+ })};
+ return name;
+}
+
std::optional<mlir::acc::DataClause>
mlir::acc::getDataClause(mlir::Operation *accDataEntryOp) {
auto dataClause{
@@ -1362,3 +1406,22 @@ bool mlir::acc::getImplicitFlag(mlir::Operation *accDataEntryOp) {
.Default([&](mlir::Operation *) { return false; })};
return implicit;
}
+
+mlir::ValueRange mlir::acc::getDataOperands(mlir::Operation *accOp) {
+ auto dataOperands{
+ llvm::TypeSwitch<mlir::Operation *, mlir::ValueRange>(accOp)
+ .Case<ACC_COMPUTE_AND_DATA_CONSTRUCT_OPS>(
+ [&](auto entry) { return entry.getDataClauseOperands(); })
+ .Default([&](mlir::Operation *) { return mlir::ValueRange(); })};
+ return dataOperands;
+}
+
+mlir::MutableOperandRange
+mlir::acc::getMutableDataOperands(mlir::Operation *accOp) {
+ auto dataOperands{
+ llvm::TypeSwitch<mlir::Operation *, mlir::MutableOperandRange>(accOp)
+ .Case<ACC_COMPUTE_AND_DATA_CONSTRUCT_OPS>(
+ [&](auto entry) { return entry.getDataClauseOperandsMutable(); })
+ .Default([&](mlir::Operation *) { return nullptr; })};
+ return dataOperands;
+}
>From bddbdcc9351ce445a941423b614de9a49d28953f Mon Sep 17 00:00:00 2001
From: Razvan Lupusoru <rlupusoru at nvidia.com>
Date: Tue, 19 Dec 2023 14:18:59 -0800
Subject: [PATCH 2/2] Fix effects of `acc.reduction`
Since this operation gets decomposed to `acc.copyin` at entry to
region, it does not need to model RuntimeCounters resource. So
model it more like `acc.firstprivate`.
---
mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
index 4abc0d62dbfdae..a78c3e98c95517 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
@@ -331,9 +331,7 @@ def OpenACC_FirstprivateOp : OpenACC_DataEntryOp<"firstprivate",
// 2.5.15 reduction clause
//===----------------------------------------------------------------------===//
def OpenACC_ReductionOp : OpenACC_DataEntryOp<"reduction",
- "mlir::acc::DataClause::acc_reduction", "",
- [MemoryEffects<[MemRead<OpenACC_RuntimeCounters>,
- MemWrite<OpenACC_RuntimeCounters>]>],
+ "mlir::acc::DataClause::acc_reduction", "", [],
(ins Arg<OpenACC_PointerLikeTypeInterface,"Address of variable",[MemRead]>:$varPtr)> {
let summary = "Represents reduction semantics for acc reduction clause.";
let results = (outs Arg<OpenACC_PointerLikeTypeInterface,
More information about the flang-commits
mailing list