[Mlir-commits] [mlir] [mlir][acc] Add data directive codegen to ACCToLLVM (PR #223043)
Razvan Lupusoru
llvmlistbot at llvm.org
Fri Sep 11 13:02:47 PDT 2026
https://github.com/razvanlupusoru created https://github.com/llvm/llvm-project/pull/223043
Extends the acc to LLVM conversion with codegen for the OpenACC data constructs - data, enter data, exit data and update. Each construct becomes calls to the libacctarget data entry points that establish the mappings its clauses ask for and tear them down again, on the queue its async clause selects and after the wait its wait clause asks for. As with the executable directives, these runtime APIs are not yet finalized; the draft proposal can be found at https://github.com/llvm/llvm-project/pull/197894.
A mapping is handed to the runtime as parallel argument arrays holding one entry per mapped object, together with a descriptor for the objects whose extent or layout cannot be stated as a size. The descriptor layouts are declared next to the entry points they are passed to, as they are equally part of the runtime interface.
Clause handling the data constructs share with the executable directives, such as async, wait and if, now lives in the common conversion utilities.
>From e55784d644f86d5e3342ac3c950c5aeb392f72e6 Mon Sep 17 00:00:00 2001
From: Razvan Lupusoru <rlupusoru at nvidia.com>
Date: Fri, 11 Sep 2026 12:01:07 -0700
Subject: [PATCH] [mlir][acc] Add data directive codegen to ACCToLLVM
Extends the acc to LLVM conversion with codegen for the OpenACC data
constructs - data, enter data, exit data and update. Each construct becomes
calls to the libacctarget data entry points that establish the mappings its
clauses ask for and tear them down again, on the queue its async clause
selects and after the wait its wait clause asks for. As with the executable
directives, these runtime APIs are not yet finalized; the draft proposal can
be found at https://github.com/llvm/llvm-project/pull/197894.
A mapping is handed to the runtime as parallel argument arrays holding one
entry per mapped object, together with a descriptor for the objects whose
extent or layout cannot be stated as a size. The descriptor layouts are
declared next to the entry points they are passed to, as they are equally
part of the runtime interface.
Clause handling the data constructs share with the executable directives,
such as async, wait and if, now lives in the common conversion utilities.
Co-authored-by: Matsu <kmatsumura at nvidia.com>
---
.../Conversion/OpenACCToLLVM/ACCDataRuntime.h | 121 ++++
.../mlir/Conversion/OpenACCToLLVM/ACCToLLVM.h | 37 +-
.../Conversion/OpenACCToLLVM/ACCToLLVMUtils.h | 204 ++++++-
.../OpenACC/OpenACCRuntimeDescriptors.def | 81 +++
.../OpenACC/OpenACCRuntimeFunctions.def | 45 ++
.../Dialect/OpenACC/OpenACCRuntimeUtils.h | 81 ++-
.../mlir/Dialect/OpenACC/OpenACCUtilsCG.h | 12 +
.../ACCDataDirectivePatterns.cpp | 278 ++++++++++
.../OpenACCToLLVM/ACCDataRuntime.cpp | 477 ++++++++++++++++
.../ACCExecutableDirectivePatterns.cpp | 159 ++----
.../Conversion/OpenACCToLLVM/ACCToLLVM.cpp | 13 +-
.../OpenACCToLLVM/ACCToLLVMUtils.cpp | 210 +++++--
.../Conversion/OpenACCToLLVM/CMakeLists.txt | 3 +
.../OpenACC/Utils/OpenACCRuntimeUtils.cpp | 109 +++-
.../Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp | 45 +-
.../OpenACCToLLVM/data-diagnostics.mlir | 13 +
.../OpenACCToLLVM/data-runtime.mlir | 522 ++++++++++++++++++
mlir/test/Conversion/OpenACCToLLVM/data.mlir | 343 ++++++++++++
.../OpenACCToLLVM/init-shutdown-set.mlir | 26 +-
.../runtime-declaration-mismatch.mlir | 21 +
mlir/test/Conversion/OpenACCToLLVM/wait.mlir | 16 +-
mlir/unittests/Dialect/OpenACC/CMakeLists.txt | 1 +
.../OpenACC/OpenACCRuntimeUtilsTest.cpp | 119 ++++
23 files changed, 2742 insertions(+), 194 deletions(-)
create mode 100644 mlir/include/mlir/Conversion/OpenACCToLLVM/ACCDataRuntime.h
create mode 100644 mlir/include/mlir/Dialect/OpenACC/OpenACCRuntimeDescriptors.def
create mode 100644 mlir/lib/Conversion/OpenACCToLLVM/ACCDataDirectivePatterns.cpp
create mode 100644 mlir/lib/Conversion/OpenACCToLLVM/ACCDataRuntime.cpp
create mode 100644 mlir/test/Conversion/OpenACCToLLVM/data-diagnostics.mlir
create mode 100644 mlir/test/Conversion/OpenACCToLLVM/data-runtime.mlir
create mode 100644 mlir/test/Conversion/OpenACCToLLVM/data.mlir
create mode 100644 mlir/unittests/Dialect/OpenACC/OpenACCRuntimeUtilsTest.cpp
diff --git a/mlir/include/mlir/Conversion/OpenACCToLLVM/ACCDataRuntime.h b/mlir/include/mlir/Conversion/OpenACCToLLVM/ACCDataRuntime.h
new file mode 100644
index 0000000000000..7ce82b5339866
--- /dev/null
+++ b/mlir/include/mlir/Conversion/OpenACCToLLVM/ACCDataRuntime.h
@@ -0,0 +1,121 @@
+//===- ACCDataRuntime.h - OpenACC data runtime arguments --------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Arguments and descriptors the OpenACC data runtime entry points are called
+// with. A conversion that lowers its own construct to those entry points emits
+// them with these, rather than restating the argument layout.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_CONVERSION_OPENACCTOLLVM_ACCDATARUNTIME_H
+#define MLIR_CONVERSION_OPENACCTOLLVM_ACCDATARUNTIME_H
+
+#include "mlir/Dialect/OpenACC/OpenACCRuntimeUtils.h"
+#include "mlir/IR/Value.h"
+#include "llvm/ADT/SmallVector.h"
+
+#include <cstdint>
+
+namespace mlir {
+class MemRefType;
+class Operation;
+class ConversionPatternRewriter;
+class RewriterBase;
+class Region;
+class SymbolTable;
+
+namespace acc {
+class OpenACCSupport;
+} // namespace acc
+
+/// The arguments every mapping entry point of the OpenACC runtime takes, in the
+/// order they are passed. An entry point that maps `argNum` objects reads one
+/// element from each of the arrays below per object; an entry point may also
+/// take further arguments of its own, which are not part of this set.
+struct ACCDataRuntimeArgs {
+ /// Source position and enclosing function name of the directive.
+ Value ident;
+ /// Reserved for per-call runtime flags; no flag is defined yet.
+ Value flags;
+ /// Device type the directive applies to, in the runtime encoding.
+ Value deviceType;
+ /// Number of mapped objects, that is, the length of each array below.
+ Value argNum;
+ /// Array of pointer slots to attach a mapped object to, null when the object
+ /// is not attached to anything.
+ Value argBasePtrs;
+ /// Array of addresses of the mapped objects themselves.
+ Value argPtrs;
+ /// Array of object sizes in bytes; zero where the descriptor or the bounds
+ /// state the extent instead.
+ Value argSizes;
+ /// Array of map-type flags, in the runtime encoding of `acc::MapFlags`.
+ Value argTypes;
+ /// Array of variable names for runtime diagnostics, null where unknown.
+ Value argNames;
+ /// Array of user-defined mappers. OpenACC has none, so this is always null.
+ Value argMappers;
+ /// Array of descriptors stating layout and bounds, null for whole objects of
+ /// a known size.
+ Value argDescs;
+
+ /// Returns the fields above in the order the entry points take them, so that
+ /// a caller only appends the arguments specific to the one it calls.
+ SmallVector<Value> getCallArgs() const;
+};
+
+/// Identifies how data runtime arguments will be consumed.
+enum class ACCDataCallKind {
+ DataEnter,
+ DataExit,
+};
+
+/// Materialize \p values as a stack-allocated LLVM array.
+Value createACCDataArray(Location loc, Type elementType, ArrayRef<Value> values,
+ RewriterBase &rewriter);
+
+/// Wrap \p baseDescriptor in the OpenACC overlay that carries \p bounds.
+/// Field zero of \p baseDescriptor is set to \p version with the OpenACC
+/// descriptor bit added.
+Value createACCDataDescriptor(Location loc, Value baseDescriptor,
+ Type baseDescriptorType, uint32_t version,
+ ValueRange bounds, Value elementSize,
+ ConversionPatternRewriter &rewriter);
+
+/// Build the runtime argument descriptor wrapping an already converted memref.
+/// The memref descriptor itself is the one the memref-to-LLVM conversion
+/// produced; this points the runtime at it. When \p bounds is non-empty, the
+/// wrapper is nested in the OpenACC overlay that carries those bounds instead
+/// of being stored on its own.
+Value createACCMemRefDescriptorWrapperArg(Location loc, MemRefType memrefType,
+ Value convertedMemref,
+ ValueRange bounds, Value elementSize,
+ ConversionPatternRewriter &rewriter);
+
+/// Build the runtime argument descriptor for \p mapOp, or a null pointer when
+/// the mapping needs no descriptor.
+Value createACCArgumentDescriptor(Operation *mapOp, Value convertedOperand,
+ acc::OpenACCSupport &accSupport,
+ ConversionPatternRewriter &rewriter);
+
+/// Emit the OpenACC data runtime arguments for data-clause operands. The
+/// operands may be `acc.map_info` or the data-clause operations it replaces.
+/// The emitted arrays name globals after the mapped objects and the position
+/// of the directive; they are created in \p globalSymbolRegion, with \p
+/// symbolTable as in acc::getOrCreateGlobalString.
+LogicalResult emitACCDataRuntimeArgs(
+ Location loc, ValueRange mappingOperands, ValueRange convertedOperands,
+ ConversionPatternRewriter &rewriter, Region &globalSymbolRegion,
+ acc::OpenACCSupport &accSupport, const acc::ACCRuntimeCallConfig &config,
+ ACCDataRuntimeArgs &runtimeArgs,
+ ACCDataCallKind callKind = ACCDataCallKind::DataEnter,
+ SymbolTable *symbolTable = nullptr);
+
+} // namespace mlir
+
+#endif // MLIR_CONVERSION_OPENACCTOLLVM_ACCDATARUNTIME_H
diff --git a/mlir/include/mlir/Conversion/OpenACCToLLVM/ACCToLLVM.h b/mlir/include/mlir/Conversion/OpenACCToLLVM/ACCToLLVM.h
index 6e69aca0f1c71..039a513844936 100644
--- a/mlir/include/mlir/Conversion/OpenACCToLLVM/ACCToLLVM.h
+++ b/mlir/include/mlir/Conversion/OpenACCToLLVM/ACCToLLVM.h
@@ -10,6 +10,8 @@
#define MLIR_CONVERSION_OPENACCTOLLVM_ACCTOLLVM_H
#include "mlir/Dialect/OpenACC/OpenACCRuntimeUtils.h"
+#include "mlir/IR/BuiltinOps.h"
+#include "mlir/IR/Value.h"
#include <functional>
#include <memory>
@@ -20,7 +22,8 @@ class LLVMTypeConverter;
class Operation;
class Pass;
class RewritePatternSet;
-class Value;
+class Region;
+class SymbolTable;
namespace acc {
class OpenACCSupport;
@@ -35,9 +38,12 @@ void configureACCExecutableDirectiveConversionLegality(
ConversionTarget &target);
/// Populate patterns that lower OpenACC executable directives (init, shutdown,
-/// wait, set) to LLVM runtime calls.
+/// wait, set) to LLVM runtime calls. The runtime declarations and globals the
+/// patterns add are created in \p globalSymbolRegion and registered in
+/// \p symbolTable.
void populateACCExecutableDirectivePatterns(
LLVMTypeConverter &converter, RewritePatternSet &patterns,
+ Region &globalSymbolRegion, SymbolTable &symbolTable,
const acc::ACCRuntimeCallConfig &config = {});
/// Returns the address operand of a dialect-specific load operation. Used when
@@ -55,6 +61,33 @@ void populateACCAtomicPatterns(
acc::OpenACCSupport &accSupport,
ACCAtomicLoadAddressCallback getLoadAddress = {});
+/// Configure conversion legality for OpenACC data directives.
+void configureACCDataDirectiveConversionLegality(ConversionTarget &target);
+
+/// Populate patterns that lower OpenACC data directives (`acc.data`,
+/// `enter_data`, `exit_data`, `update`) to `__tgt_acc_data_*` runtime calls.
+/// Clauses that can be repeated per device type are taken from the ones that
+/// apply to \p clauseDeviceType. The runtime declarations and globals the
+/// patterns add are created in \p globalSymbolRegion and registered in
+/// \p symbolTable.
+void populateACCDataDirectivePatterns(
+ LLVMTypeConverter &converter, RewritePatternSet &patterns,
+ acc::OpenACCSupport &accSupport, Region &globalSymbolRegion,
+ SymbolTable &symbolTable, const acc::ACCRuntimeCallConfig &config = {},
+ acc::DeviceType clauseDeviceType = acc::DeviceType::None);
+
+/// Populate the patterns that remove OpenACC data clause operations once the
+/// constructs holding them have turned their mappings into runtime calls. A
+/// data entry operation is replaced by the address of the object it named, and
+/// a data exit or bounds operation is erased. Clause operations whose result
+/// is not that address stay with the construct that holds them.
+///
+/// A conversion has to populate these only once every construct holding such a
+/// clause operation is lowered in the same conversion, as the mappings would
+/// otherwise be lost.
+void populateACCDataClauseOpPatterns(LLVMTypeConverter &converter,
+ RewritePatternSet &patterns);
+
} // namespace mlir
#endif // MLIR_CONVERSION_OPENACCTOLLVM_ACCTOLLVM_H
diff --git a/mlir/include/mlir/Conversion/OpenACCToLLVM/ACCToLLVMUtils.h b/mlir/include/mlir/Conversion/OpenACCToLLVM/ACCToLLVMUtils.h
index 7e16cf17c4489..73d4ca156b9e0 100644
--- a/mlir/include/mlir/Conversion/OpenACCToLLVM/ACCToLLVMUtils.h
+++ b/mlir/include/mlir/Conversion/OpenACCToLLVM/ACCToLLVMUtils.h
@@ -9,14 +9,22 @@
#ifndef MLIR_CONVERSION_OPENACCTOLLVM_ACCTOLLVMUTILS_H
#define MLIR_CONVERSION_OPENACCTOLLVM_ACCTOLLVMUTILS_H
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
#include "mlir/Dialect/OpenACC/OpenACCRuntimeUtils.h"
#include "mlir/IR/Builders.h"
#include "mlir/IR/BuiltinOps.h"
#include "mlir/IR/Location.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/Region.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/STLForwardCompat.h"
+#include "llvm/ADT/STLFunctionalExtras.h"
#include "llvm/ADT/StringRef.h"
#include <optional>
#include <string>
+#include <utility>
namespace mlir {
namespace acc {
@@ -28,7 +36,8 @@ Location unfuseLoc(Location loc);
std::optional<FileLineColLoc> getFileLineColLoc(Location loc,
bool errorOnInvalidLocation);
-/// Returns the enclosing function symbol name for \p op.
+/// Returns the symbol name of the function \p op belongs to, or of \p op itself
+/// when it is a function.
StringRef getParentFunctionName(Operation *op);
/// Returns the enclosing function symbol name for \p value's defining op.
@@ -37,14 +46,199 @@ StringRef getParentFunctionName(Value value);
/// Returns the first non-empty enclosing function name from \p values.
StringRef getParentFunctionName(ValueRange values);
-/// Creates or reuses a module-internal null-terminated string global.
+/// Returns the name to give a global that the conversion creates to hold
+/// \p detail of \p kind, such as the name of a variable or a source position.
+/// The dots make a name no identifier of the program can carry, so these
+/// globals are reachable by name without walking the symbols of the module.
+/// \p detail keeps letters, digits, `_`, `$` and `.` and folds every other
+/// character to an underscore. Distinct details can therefore collide on the
+/// same name; getOrCreateGlobalString is what then keeps the globals apart.
+std::string getInternalGlobalName(StringRef kind, StringRef detail);
+
+/// Creates or reuses a null-terminated string global in \p globalSymbolRegion.
+/// With \p symbolTable, a global already going by \p name is reused when it
+/// holds \p value, and a name that two different values arrive under gets a
+/// suffix to tell the globals apart. Without it the global is created under
+/// \p name as given, which only a caller whose names are unique by
+/// construction can ask for.
Value getOrCreateGlobalString(Location loc, OpBuilder &builder, StringRef name,
- StringRef value, ModuleOp module);
+ StringRef value, Region &globalSymbolRegion,
+ SymbolTable *symbolTable = nullptr);
/// Returns a pointer to a constant global holding an ident_t for OpenACC
-/// runtime calls.
+/// runtime calls. \p globalSymbolRegion and \p symbolTable are as in
+/// getOrCreateGlobalString; the ident and the source string it points to are
+/// named after the position they describe, so leaving out the table creates a
+/// set of them per call.
Value createIdent(Location loc, StringRef functionName, OpBuilder &builder,
- ModuleOp module, const ACCRuntimeCallConfig &config);
+ Region &globalSymbolRegion,
+ const ACCRuntimeCallConfig &config,
+ SymbolTable *symbolTable = nullptr);
+
+/// Sign-extends or truncates \p value to the i64 the runtime entry points take
+/// for values like queue numbers.
+Value castToI64(Location loc, Value value, OpBuilder &builder);
+
+/// Returns the queue an `async` clause selects: the value of the clause when it
+/// has one, the queue standing for an `async` clause without a value when
+/// \p asyncOnly is set, and the synchronous queue when there is no clause at
+/// all. \p asyncOperand must already be converted to the LLVM dialect.
+Value getAsyncQueue(Location loc, Value asyncOperand, bool asyncOnly,
+ OpBuilder &builder, const ACCRuntimeCallConfig &config);
+
+/// Emits the runtime call that waits for \p waitOperands on \p asyncQueue,
+/// which is what a `wait` clause or an `acc.wait` directive asks for. An empty
+/// \p waitOperands waits for every queue, as a `wait` clause without values
+/// does. The values must already be converted to the LLVM dialect.
+LogicalResult emitWaitCall(Location loc, ValueRange waitOperands,
+ Value asyncQueue, OpBuilder &builder,
+ Region &globalSymbolRegion, SymbolTable &symbolTable,
+ const ACCRuntimeCallConfig &config);
+
+/// Runs \p emitFn guarded by a branch on \p ifCond, or unguarded when there is
+/// no condition. Leaves the insertion point after the guarded code, so that a
+/// caller can keep emitting into the same block either way.
+LogicalResult emitGuardedByIfCond(Location loc, Value ifCond,
+ RewriterBase &rewriter,
+ function_ref<LogicalResult()> emitFn);
+
+/// The clauses of a construct can be given once per device type. Of the values
+/// that reach a given device type, the ones naming it are the most specific,
+/// then the ones naming every device type, then the ones given before any
+/// device_type clause.
+SmallVector<DeviceType, 3> getDeviceTypesByPrecedence(DeviceType deviceType);
+
+namespace detail {
+/// The constructs carrying a `device_type` clause hold their async and wait
+/// clauses per device type. The others, such as `acc.enter_data` or
+/// `acc.kernel_environment`, hold a single value for each clause, spelling the
+/// value-less form either `asyncOnly`/`waitOnly` or `async`/`wait`.
+template <typename OpTy>
+using has_device_type_clauses_t =
+ decltype(std::declval<OpTy>().hasAsyncOnly(DeviceType::None));
+template <typename OpTy>
+using has_async_only_t = decltype(std::declval<OpTy>().getAsyncOnly());
+template <typename OpTy>
+using has_wait_only_t = decltype(std::declval<OpTy>().getWaitOnly());
+} // namespace detail
+
+/// Returns the value the `async` clause of \p op names for \p deviceType, and
+/// sets \p asyncOnly when the clause names no queue. The value is the one the
+/// operation holds, so a caller in a conversion has to remap it.
+template <typename OpTy>
+Value getAsyncClauseValue(OpTy op, DeviceType deviceType, bool &asyncOnly) {
+ asyncOnly = false;
+ if constexpr (llvm::is_detected<detail::has_device_type_clauses_t,
+ OpTy>::value) {
+ for (DeviceType candidate : getDeviceTypesByPrecedence(deviceType)) {
+ if (op.hasAsyncOnly(candidate)) {
+ asyncOnly = true;
+ return {};
+ }
+ if (Value asyncValue = op.getAsyncValue(candidate))
+ return asyncValue;
+ }
+ return {};
+ } else if constexpr (llvm::is_detected<detail::has_async_only_t,
+ OpTy>::value) {
+ asyncOnly = op.getAsyncOnly();
+ return op.getAsyncOperand();
+ } else {
+ asyncOnly = op.getAsync();
+ return op.getAsyncOperand();
+ }
+}
+
+/// Appends to \p waitValues the queues the `wait` clause of \p op names for
+/// \p deviceType, and returns the device type the clause naming them is given
+/// for - a clause naming no queue waits for every one of them. Returns
+/// std::nullopt when \p op gives no such clause for \p deviceType. The values
+/// are the ones the operation holds, so a caller in a conversion has to remap
+/// them.
+template <typename OpTy>
+std::optional<DeviceType>
+getWaitClauseValues(OpTy op, DeviceType deviceType,
+ SmallVectorImpl<Value> &waitValues) {
+ if constexpr (llvm::is_detected<detail::has_device_type_clauses_t,
+ OpTy>::value) {
+ for (DeviceType candidate : getDeviceTypesByPrecedence(deviceType)) {
+ if (op.hasWaitOnly(candidate))
+ return candidate;
+ auto values = op.getWaitValues(candidate);
+ if (!values.empty()) {
+ llvm::append_range(waitValues, values);
+ return candidate;
+ }
+ }
+ return std::nullopt;
+ } else {
+ llvm::append_range(waitValues, op.getWaitOperands());
+ bool waitsForEveryQueue;
+ if constexpr (llvm::is_detected<detail::has_wait_only_t, OpTy>::value)
+ waitsForEveryQueue = op.getWaitOnly();
+ else
+ waitsForEveryQueue = op.getWait();
+ if (!waitsForEveryQueue && waitValues.empty())
+ return std::nullopt;
+ // The operation holds a single wait clause, which no device_type clause
+ // narrows to a device type.
+ return DeviceType::None;
+ }
+}
+
+/// Returns whether the `wait` clause \p op gives for \p deviceType carries a
+/// devnum modifier, which selects the device the queues belong to. Only the
+/// clause given for that device type is asked, so a caller passes the device
+/// type getWaitClauseValues took the queues from.
+template <typename OpTy>
+bool hasWaitDevnum(OpTy op, DeviceType deviceType) {
+ if constexpr (llvm::is_detected<detail::has_device_type_clauses_t,
+ OpTy>::value) {
+ return static_cast<bool>(op.getWaitDevnum(deviceType));
+ } else {
+ return static_cast<bool>(op.getWaitDevnum());
+ }
+}
+
+/// Returns the queue that the runtime calls of \p op run on, from the `async`
+/// clause it gives for \p deviceType.
+template <typename OpTy>
+Value getAsyncQueue(OpTy op, DeviceType deviceType,
+ ConversionPatternRewriter &rewriter,
+ const ACCRuntimeCallConfig &config) {
+ bool asyncOnly = false;
+ Value asyncValue = getAsyncClauseValue(op, deviceType, asyncOnly);
+ if (asyncValue)
+ asyncValue = rewriter.getRemappedValue(asyncValue);
+ return getAsyncQueue(op.getLoc(), asyncValue, asyncOnly, rewriter, config);
+}
+
+/// Emits the wait that a `wait` clause on \p op asks for before the runtime
+/// calls of the construct, waiting on \p asyncQueue for the queues the clause
+/// names, or for every queue when it names none. Nothing is emitted when there
+/// is no such clause for \p deviceType. A devnum modifier, which selects the
+/// device the queues belong to, is reported through \p accSupport as not yet
+/// implemented.
+template <typename OpTy>
+LogicalResult
+emitWaitClause(OpTy op, DeviceType deviceType, Value asyncQueue,
+ ConversionPatternRewriter &rewriter, OpenACCSupport &accSupport,
+ Region &globalSymbolRegion, SymbolTable &symbolTable,
+ const ACCRuntimeCallConfig &config) {
+ SmallVector<Value> waitValues;
+ std::optional<DeviceType> clauseDeviceType =
+ getWaitClauseValues(op, deviceType, waitValues);
+ if (!clauseDeviceType)
+ return success();
+ if (hasWaitDevnum(op, *clauseDeviceType)) {
+ (void)accSupport.emitNYI(op.getLoc(), "wait clause with a devnum modifier");
+ return failure();
+ }
+ for (Value &waitValue : waitValues)
+ waitValue = rewriter.getRemappedValue(waitValue);
+ return emitWaitCall(op.getLoc(), waitValues, asyncQueue, rewriter,
+ globalSymbolRegion, symbolTable, config);
+}
} // namespace acc
} // namespace mlir
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCRuntimeDescriptors.def b/mlir/include/mlir/Dialect/OpenACC/OpenACCRuntimeDescriptors.def
new file mode 100644
index 0000000000000..aa046bb0e1037
--- /dev/null
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCRuntimeDescriptors.def
@@ -0,0 +1,81 @@
+//===- OpenACCRuntimeDescriptors.def - ACC descriptor catalog ---*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+/// \file
+///
+/// X-macro catalog of the argument descriptors the compiler materializes for
+/// the `arg_descs` parameter of the `__tgt_acc_data_*` entry points. Field
+/// order, field types, and padding are part of the runtime ABI and must match
+/// the `AccDataDesc*` types of the offload runtime, which are not restated
+/// here. Include this file after defining ACC_DESC_BEGIN.
+///
+/// ACC_DESC_BEGIN(Enum, NameStr, DescKind)
+/// Enum - enumerator used as DataDescriptor::Enum
+/// NameStr - name of the runtime type this layout materializes
+/// DescKind - `acc::DataDescKind` the runtime reads from the version field
+///
+/// ACC_DESC_FIELD(Enum, Field, TypeToken)
+/// Field - enumerator used as Enum##Field::Field, in declaration order
+/// TypeToken - MLIR LLVM type token (Int8, Int32, Int64, Ptr), or Base for
+/// the descriptor a layout nests
+///
+/// ACC_DESC_END(Enum)
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef ACC_DESC_BEGIN
+#error "define ACC_DESC_BEGIN before including OpenACCRuntimeDescriptors.def"
+#endif
+
+#ifndef ACC_DESC_FIELD
+#define ACC_DESC_FIELD(Enum, Field, TypeToken)
+#endif
+
+#ifndef ACC_DESC_END
+#define ACC_DESC_END(Enum)
+#endif
+
+// A mapping that needs no descriptor of its own, used as the base of an
+// OpenACC overlay that only carries bounds.
+ACC_DESC_BEGIN(AccDataDescGeneric, "AccDataDescGeneric", DataDescKind::none)
+ACC_DESC_FIELD(AccDataDescGeneric, Version, Int32)
+ACC_DESC_FIELD(AccDataDescGeneric, Padding, Int32)
+ACC_DESC_END(AccDataDescGeneric)
+
+// A mapping described by a Fortran descriptor (`CFI_cdesc_t` of
+// ISO_Fortran_binding.h), which the descriptor field addresses.
+ACC_DESC_BEGIN(AccDataDescCFI, "AccDataDescCFI", DataDescKind::cfi)
+ACC_DESC_FIELD(AccDataDescCFI, Version, Int32)
+ACC_DESC_FIELD(AccDataDescCFI, CFIDescriptor, Ptr)
+ACC_DESC_END(AccDataDescCFI)
+
+// A mapping described by the memref descriptor the memref-to-LLVM conversion
+// produced, which the descriptor field addresses.
+ACC_DESC_BEGIN(AccDataDescMemRef, "AccDataDescMemRef", DataDescKind::memref)
+ACC_DESC_FIELD(AccDataDescMemRef, Version, Int32)
+ACC_DESC_FIELD(AccDataDescMemRef, Rank, Int8)
+ACC_DESC_FIELD(AccDataDescMemRef, ElementSize, Int64)
+ACC_DESC_FIELD(AccDataDescMemRef, MemRefDescriptor, Ptr)
+ACC_DESC_END(AccDataDescMemRef)
+
+// Bounds of a mapping, overlaid on the descriptor that states its storage.
+// Rank is the number of bounds and each array below holds that many entries,
+// in dimension order.
+ACC_DESC_BEGIN(AccDataDescOpenACC, "AccDataDescOpenACC", DataDescKind::openacc)
+ACC_DESC_FIELD(AccDataDescOpenACC, Base, Base)
+ACC_DESC_FIELD(AccDataDescOpenACC, Rank, Int8)
+ACC_DESC_FIELD(AccDataDescOpenACC, ElementSize, Int64)
+ACC_DESC_FIELD(AccDataDescOpenACC, LowerBounds, Ptr)
+ACC_DESC_FIELD(AccDataDescOpenACC, UpperBounds, Ptr)
+ACC_DESC_FIELD(AccDataDescOpenACC, Extents, Ptr)
+ACC_DESC_FIELD(AccDataDescOpenACC, StridesInBytes, Ptr)
+ACC_DESC_FIELD(AccDataDescOpenACC, StartIndices, Ptr)
+ACC_DESC_END(AccDataDescOpenACC)
+
+#undef ACC_DESC_BEGIN
+#undef ACC_DESC_FIELD
+#undef ACC_DESC_END
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCRuntimeFunctions.def b/mlir/include/mlir/Dialect/OpenACC/OpenACCRuntimeFunctions.def
index 338f3950216f5..cffa692428b8b 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCRuntimeFunctions.def
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCRuntimeFunctions.def
@@ -53,5 +53,50 @@ __ACC_RTL(tgt_acc_set_device_num, false, Void, Ptr, Int64, Int64, Int64)
// void __tgt_acc_set_device_type(ident_t *, int64_t flags, int64_t device_type);
__ACC_RTL(tgt_acc_set_device_type, false, Void, Ptr, Int64, Int64)
+// void __tgt_acc_data_begin(ident_t *, int64_t flags, int64_t device_type,
+// uint32_t arg_num, void **arg_base_ptrs,
+// void **arg_ptrs, int64_t *arg_sizes,
+// int64_t *arg_types, char **arg_names,
+// void **arg_mappers, AccDataDesc **arg_descs,
+// int64_t async);
+__ACC_RTL(tgt_acc_data_begin, false, Void, Ptr, Int64, Int64, Int32, Ptr, Ptr,
+ Ptr, Ptr, Ptr, Ptr, Ptr, Int64)
+
+// void __tgt_acc_data_end(ident_t *, int64_t flags, int64_t device_type,
+// uint32_t arg_num, void **arg_base_ptrs,
+// void **arg_ptrs, int64_t *arg_sizes,
+// int64_t *arg_types, char **arg_names,
+// void **arg_mappers, AccDataDesc **arg_descs,
+// int64_t async);
+__ACC_RTL(tgt_acc_data_end, false, Void, Ptr, Int64, Int64, Int32, Ptr, Ptr, Ptr,
+ Ptr, Ptr, Ptr, Ptr, Int64)
+
+// void __tgt_acc_data_enter(ident_t *, int64_t flags, int64_t device_type,
+// uint32_t arg_num, void **arg_base_ptrs,
+// void **arg_ptrs, int64_t *arg_sizes,
+// int64_t *arg_types, char **arg_names,
+// void **arg_mappers, AccDataDesc **arg_descs,
+// int64_t async);
+__ACC_RTL(tgt_acc_data_enter, false, Void, Ptr, Int64, Int64, Int32, Ptr, Ptr,
+ Ptr, Ptr, Ptr, Ptr, Ptr, Int64)
+
+// void __tgt_acc_data_exit(ident_t *, int64_t flags, int64_t device_type,
+// uint32_t arg_num, void **arg_base_ptrs,
+// void **arg_ptrs, int64_t *arg_sizes,
+// int64_t *arg_types, char **arg_names,
+// void **arg_mappers, AccDataDesc **arg_descs,
+// int64_t async);
+__ACC_RTL(tgt_acc_data_exit, false, Void, Ptr, Int64, Int64, Int32, Ptr, Ptr,
+ Ptr, Ptr, Ptr, Ptr, Ptr, Int64)
+
+// void __tgt_acc_data_update(ident_t *, int64_t flags, int64_t device_type,
+// uint32_t arg_num, void **arg_base_ptrs,
+// void **arg_ptrs, int64_t *arg_sizes,
+// int64_t *arg_types, char **arg_names,
+// void **arg_mappers, AccDataDesc **arg_descs,
+// int64_t async);
+__ACC_RTL(tgt_acc_data_update, false, Void, Ptr, Int64, Int64, Int32, Ptr, Ptr,
+ Ptr, Ptr, Ptr, Ptr, Ptr, Int64)
+
#undef __ACC_RTL
#undef ACC_RTL
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCRuntimeUtils.h b/mlir/include/mlir/Dialect/OpenACC/OpenACCRuntimeUtils.h
index 4274bacd974eb..93662cb1a5938 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCRuntimeUtils.h
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCRuntimeUtils.h
@@ -18,6 +18,7 @@
#include "mlir/Dialect/OpenACC/OpenACC.h"
#include "mlir/IR/Builders.h"
#include "mlir/IR/BuiltinOps.h"
+#include "mlir/IR/Region.h"
#include "llvm/ADT/ArrayRef.h"
#include "llvm/ADT/DenseMap.h"
#include "llvm/ADT/StringRef.h"
@@ -42,11 +43,48 @@ StringRef getRuntimeFunctionName(RuntimeFunction fn);
LLVM::LLVMFunctionType getRuntimeFunctionType(MLIRContext *ctx,
RuntimeFunction fn);
-/// Optional overrides for OpenACC to LLVM runtime lowering.
+/// IDs for the argument descriptors of the OpenACC data entry points, declared
+/// in OpenACCRuntimeDescriptors.def.
+enum class DataDescriptor {
+#define ACC_DESC_BEGIN(Enum, ...) Enum,
+#include "mlir/Dialect/OpenACC/OpenACCRuntimeDescriptors.def"
+};
+
+/// Field indices of each descriptor, in declaration order.
+#define ACC_DESC_BEGIN(Enum, ...) enum class Enum##Field : int64_t {
+#define ACC_DESC_FIELD(Enum, Field, TypeToken) Field,
+#define ACC_DESC_END(Enum) \
+ } \
+ ;
+#include "mlir/Dialect/OpenACC/OpenACCRuntimeDescriptors.def"
+
+/// The index an insert or extract of \p field addresses.
+template <typename FieldEnum>
+int64_t getDataDescriptorFieldIndex(FieldEnum field) {
+ return static_cast<int64_t>(field);
+}
+
+/// Returns the name of the runtime type \p desc materializes.
+StringRef getDataDescriptorName(DataDescriptor desc);
+
+/// Returns the descriptor kind the runtime reads from the version field of
+/// \p desc. An overlay contributes its kind to the descriptor it nests.
+DataDescKind getDataDescriptorKind(DataDescriptor desc);
+
+/// Builds the LLVM type of \p desc in \p ctx. A descriptor that nests another
+/// one takes it as \p baseType, which is ignored otherwise.
+LLVM::LLVMStructType getDataDescriptorType(MLIRContext *ctx,
+ DataDescriptor desc,
+ Type baseType = {});
+
+/// Configuration for OpenACC to LLVM runtime lowering. Device types and map
+/// flags use the OpenACC dialect encodings by default.
class ACCRuntimeCallConfig {
public:
using FunctionDisplayNameFn = std::function<std::string(StringRef)>;
+ ACCRuntimeCallConfig();
+
void setName(RuntimeFunction fn, StringRef name);
StringRef getName(RuntimeFunction fn) const;
@@ -55,11 +93,34 @@ class ACCRuntimeCallConfig {
/// Map an OpenACC dialect \p DeviceType to the integer encoding expected by
/// the target runtime. Dialect ordinals and runtime ABI values are not
- /// required to match; callers must install a mapping that matches their
- /// runtime. Querying an unmapped type is an error.
+ /// required to match. A target whose ABI differs from the default dialect
+ /// encoding must install its mapping here. Querying an unmapped type is an
+ /// error.
void setDeviceTypeRuntimeValue(DeviceType type, int64_t runtimeValue);
int64_t getDeviceTypeRuntimeValue(DeviceType type) const;
+ /// Map a single OpenACC dialect \p MapFlags bit to the bit the target runtime
+ /// gives the same meaning. getMapFlagsRuntimeValue combines the bits of a set
+ /// of flags into the encoding the runtime reads from an argument-type slot.
+ /// As with device types, dialect and runtime encodings are not required to
+ /// match. A target whose ABI differs from the default dialect encoding must
+ /// install its mapping here, and a set bit with no mapping is an error.
+ void setMapFlagRuntimeValue(MapFlags flag, int64_t runtimeValue);
+ int64_t getMapFlagsRuntimeValue(MapFlags flags) const;
+
+ /// Adjust the flags of a mapping before they are encoded. The hook is keyed
+ /// on the operation that states the mapping and is applied to every mapped
+ /// object before anything is derived from the flags. The flags are used as
+ /// computed when no hook is installed.
+ using MapFlagsPostProcessFn =
+ std::function<MapFlags(Operation *mapOp, MapFlags flags)>;
+ void setMapFlagsPostProcessFn(MapFlagsPostProcessFn fn);
+ MapFlags postProcessMapFlags(Operation *mapOp, MapFlags flags) const;
+
+ /// Renders \p flags for a diagnostic as the names of the set bits and the
+ /// decimal and hexadecimal encoding the runtime reads.
+ std::string formatMapFlags(MapFlags flags) const;
+
/// Runtime encoding of `acc_async_sync`, used when an operation carries no
/// `async` clause. OpenACC defines the name of this queue but leaves its
/// value to the implementation, so it is part of the runtime ABI.
@@ -74,7 +135,9 @@ class ACCRuntimeCallConfig {
private:
DenseMap<RuntimeFunction, std::string> overrides;
DenseMap<DeviceType, int64_t> deviceTypeRuntimeValues;
+ DenseMap<MapFlags, int64_t> mapFlagRuntimeValues;
FunctionDisplayNameFn functionDisplayNameFn;
+ MapFlagsPostProcessFn mapFlagsPostProcessFn;
// Default to the encodings used by openacc.h (`acc_async_sync` /
// `acc_async_noval`).
int64_t asyncSyncRuntimeValue = -1;
@@ -87,12 +150,20 @@ class ACCRuntimeCallConfig {
/// own mapping via \c setDeviceTypeRuntimeValue.
void populateDialectIdentityDeviceTypeMapping(ACCRuntimeCallConfig &config);
+/// Install a map-flag mapping that uses the OpenACC dialect bit positions as
+/// the runtime encoding, with the same caveat as
+/// \c populateDialectIdentityDeviceTypeMapping.
+void populateDialectIdentityMapFlagsMapping(ACCRuntimeCallConfig &config);
+
/// Declares (if needed) and returns a call to the runtime function identified
/// by \p fn using the name from \p config. Fails and emits a diagnostic if the
/// symbol is already declared with a signature the runtime cannot be called
-/// through.
+/// through. The declaration is created in \p globalSymbolRegion and
+/// registered in \p symbolTable.
FailureOr<LLVM::CallOp> createRuntimeCall(Location loc, OpBuilder &builder,
- ModuleOp module, RuntimeFunction fn,
+ Region &globalSymbolRegion,
+ SymbolTable &symbolTable,
+ RuntimeFunction fn,
const ACCRuntimeCallConfig &config,
ArrayRef<Value> arguments);
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
index 2e9a734acecf3..bb0af06c01421 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
@@ -232,6 +232,13 @@ bool hasCopyOutSibling(Operation *entryOp);
/// \p entryResult, which take it as their `accVar`.
SmallVector<Operation *> getPairedDataExitOps(Value entryResult);
+/// Returns where the mappings of \p dataClauseOperands end, taken from the
+/// first of them that says. This is where a structured construct tears its
+/// mappings down, which is past the end of its region and therefore not a
+/// position the construct itself carries. A mapping that is never closed has
+/// none.
+std::optional<Location> getMappingExitLoc(ValueRange dataClauseOperands);
+
/// Compute total mapped byte size for `acc.map_info`.
/// Returns 0 when bounds or a non-`none` descriptor kind carry size, the
/// mappable size when statically known, and -1 when the size cannot be
@@ -241,6 +248,11 @@ int64_t computeMapInfoSizeBytes(Value var, Type varType, DataDescKind descKind,
ValueRange bounds, const DataLayout &dataLayout,
OpenACCSupport *support = nullptr);
+/// Same as above, obtaining \p dataLayout from the module \p var lives in.
+int64_t computeMapInfoSizeBytes(Value var, Type varType, DataDescKind descKind,
+ ValueRange bounds,
+ OpenACCSupport *support = nullptr);
+
/// Record known extents of the source array on bounds that may describe a
/// section. Dynamic / unknown extents are left unset.
void populateSourceExtents(ValueRange bounds, ArrayRef<int64_t> shape,
diff --git a/mlir/lib/Conversion/OpenACCToLLVM/ACCDataDirectivePatterns.cpp b/mlir/lib/Conversion/OpenACCToLLVM/ACCDataDirectivePatterns.cpp
new file mode 100644
index 0000000000000..890c00b4b69d9
--- /dev/null
+++ b/mlir/lib/Conversion/OpenACCToLLVM/ACCDataDirectivePatterns.cpp
@@ -0,0 +1,278 @@
+//===- ACCDataDirectivePatterns.cpp - ACC data to LLVM ----------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Lowering of the OpenACC data constructs - data, enter data, exit data and
+// update - to the data entry points of the OpenACC runtime. Each construct
+// becomes one call per point where it establishes or tears down its mappings,
+// which carries the argument arrays its operands are packed into, the queue
+// its async clause selects and, before them, the wait its wait clause asks
+// for. An if clause guards the calls it applies to, and the body of a
+// structured construct is left in place of the construct.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Conversion/OpenACCToLLVM/ACCDataRuntime.h"
+#include "mlir/Conversion/OpenACCToLLVM/ACCToLLVM.h"
+#include "mlir/Conversion/OpenACCToLLVM/ACCToLLVMUtils.h"
+
+#include "mlir/Conversion/LLVMCommon/Pattern.h"
+#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+
+using namespace mlir;
+using namespace mlir::acc;
+
+/// Emits the call to the mapping entry point \p fn for \p mappingOperands, the
+/// data-clause operands of a data construct, paired with their type converted
+/// values in \p convertedOperands.
+static LogicalResult emitDataRuntimeCall(
+ Location loc, RuntimeFunction fn, ValueRange mappingOperands,
+ ValueRange convertedOperands, Value asyncQueue, ACCDataCallKind callKind,
+ ConversionPatternRewriter &rewriter, OpenACCSupport &accSupport,
+ Region &globalSymbolRegion, SymbolTable &symbolTable,
+ const ACCRuntimeCallConfig &config) {
+ if (mappingOperands.empty())
+ return success();
+
+ Operation *firstMapOp = mappingOperands.front().getDefiningOp();
+ ACCDataRuntimeArgs runtimeArgs;
+ if (failed(emitACCDataRuntimeArgs(
+ loc, mappingOperands, convertedOperands, rewriter, globalSymbolRegion,
+ accSupport, config, runtimeArgs, callKind, &symbolTable)))
+ return rewriter.notifyMatchFailure(
+ firstMapOp, "unsupported OpenACC data-clause operand");
+
+ SmallVector<Value> arguments = runtimeArgs.getCallArgs();
+ arguments.push_back(asyncQueue);
+ return createRuntimeCall(loc, rewriter, globalSymbolRegion, symbolTable, fn,
+ config, arguments);
+}
+
+/// Splices \p region, the body of a structured data construct, into the block
+/// holding \p op, so that the construct itself can be erased. The terminators
+/// of the region branch to the code that follows \p op.
+static void spliceDataRegion(Operation *op, Region ®ion,
+ RewriterBase &rewriter) {
+ Block *prev = op->getBlock();
+ Block *succ = rewriter.splitBlock(prev, Block::iterator(op));
+ rewriter.setInsertionPointToEnd(prev);
+ LLVM::BrOp::create(rewriter, op->getLoc(), ®ion.getBlocks().front());
+ for (Block &block : region.getBlocks()) {
+ Operation *terminator = block.getTerminator();
+ if (isa<acc::TerminatorOp>(terminator)) {
+ rewriter.setInsertionPoint(terminator);
+ LLVM::BrOp::create(rewriter, op->getLoc(), succ);
+ rewriter.eraseOp(terminator);
+ }
+ }
+ rewriter.inlineRegionBefore(region, succ);
+}
+
+namespace {
+template <typename OpTy>
+struct ACCDataDirectivePattern : public ConvertOpToLLVMPattern<OpTy> {
+ ACCDataDirectivePattern(const LLVMTypeConverter &converter,
+ OpenACCSupport &accSupport,
+ Region &globalSymbolRegion, SymbolTable &symbolTable,
+ const ACCRuntimeCallConfig &config,
+ DeviceType clauseDeviceType,
+ PatternBenefit benefit = 1)
+ : ConvertOpToLLVMPattern<OpTy>(converter, benefit),
+ accSupport(accSupport), globalSymbolRegion(globalSymbolRegion),
+ symbolTable(symbolTable), config(config),
+ clauseDeviceType(clauseDeviceType) {}
+
+ /// Returns the queue that the mapping calls of \p op run on.
+ Value getAsyncQueue(OpTy op, ConversionPatternRewriter &rewriter) const {
+ return acc::getAsyncQueue(op, clauseDeviceType, rewriter, config);
+ }
+
+ /// Emits the wait that a wait clause on \p op asks for before its mapping
+ /// calls.
+ LogicalResult emitWaitClause(OpTy op, Value asyncQueue,
+ ConversionPatternRewriter &rewriter) const {
+ return acc::emitWaitClause(op, clauseDeviceType, asyncQueue, rewriter,
+ accSupport, globalSymbolRegion, symbolTable,
+ config);
+ }
+
+ /// Emits one mapping call for the data-clause operands of \p op.
+ LogicalResult emitMappingCall(OpTy op, ValueRange convertedOperands,
+ Location loc, RuntimeFunction fn,
+ ACCDataCallKind callKind, Value asyncQueue,
+ ConversionPatternRewriter &rewriter) const {
+ return emitDataRuntimeCall(loc, fn, op.getDataClauseOperands(),
+ convertedOperands, asyncQueue, callKind,
+ rewriter, accSupport, globalSymbolRegion,
+ symbolTable, config);
+ }
+
+ OpenACCSupport &accSupport;
+ Region &globalSymbolRegion;
+ SymbolTable &symbolTable;
+ ACCRuntimeCallConfig config;
+ DeviceType clauseDeviceType;
+};
+
+struct DataOpLowering : public ACCDataDirectivePattern<DataOp> {
+ using ACCDataDirectivePattern<DataOp>::ACCDataDirectivePattern;
+
+ LogicalResult
+ matchAndRewrite(DataOp op, DataOp::Adaptor adaptor,
+ ConversionPatternRewriter &rewriter) const override {
+ // The mappings are established where the clause operations are, and torn
+ // down where the exit operations of the construct were.
+ ValueRange mappingOperands = op.getDataClauseOperands();
+ Location beginLoc = mappingOperands.empty()
+ ? op.getLoc()
+ : mappingOperands.front().getLoc();
+ Location endLoc =
+ acc::getMappingExitLoc(mappingOperands).value_or(beginLoc);
+ Value asyncQueue = getAsyncQueue(op, rewriter);
+ if (failed(emitWaitClause(op, asyncQueue, rewriter)))
+ return failure();
+
+ auto emitBegin = [&]() {
+ return emitMappingCall(op, adaptor.getDataClauseOperands(), beginLoc,
+ RuntimeFunction::ACCRTL_tgt_acc_data_begin,
+ ACCDataCallKind::DataEnter, asyncQueue, rewriter);
+ };
+ if (failed(acc::emitGuardedByIfCond(beginLoc, adaptor.getIfCond(), rewriter,
+ emitBegin)))
+ return failure();
+
+ rewriter.setInsertionPointAfter(op);
+ auto emitEnd = [&]() {
+ return emitMappingCall(op, adaptor.getDataClauseOperands(), endLoc,
+ RuntimeFunction::ACCRTL_tgt_acc_data_end,
+ ACCDataCallKind::DataExit, asyncQueue, rewriter);
+ };
+ if (failed(acc::emitGuardedByIfCond(endLoc, adaptor.getIfCond(), rewriter,
+ emitEnd)))
+ return failure();
+
+ spliceDataRegion(op, op.getRegion(), rewriter);
+ rewriter.eraseOp(op);
+ return success();
+ }
+};
+
+/// Lowering shared by the directives that map their operands with a single
+/// call: enter data, exit data and update.
+template <typename OpTy, RuntimeFunction fn, ACCDataCallKind callKind>
+struct ACCDataCallLowering : public ACCDataDirectivePattern<OpTy> {
+ using ACCDataDirectivePattern<OpTy>::ACCDataDirectivePattern;
+
+ LogicalResult
+ matchAndRewrite(OpTy op, typename OpTy::Adaptor adaptor,
+ ConversionPatternRewriter &rewriter) const override {
+ Location loc = op.getLoc();
+ Value asyncQueue = this->getAsyncQueue(op, rewriter);
+ if (failed(this->emitWaitClause(op, asyncQueue, rewriter)))
+ return failure();
+
+ auto emit = [&]() {
+ return this->emitMappingCall(op, adaptor.getDataClauseOperands(), loc, fn,
+ callKind, asyncQueue, rewriter);
+ };
+ if (failed(
+ acc::emitGuardedByIfCond(loc, adaptor.getIfCond(), rewriter, emit)))
+ return failure();
+ rewriter.eraseOp(op);
+ return success();
+ }
+};
+
+using EnterDataOpLowering =
+ ACCDataCallLowering<EnterDataOp, RuntimeFunction::ACCRTL_tgt_acc_data_enter,
+ ACCDataCallKind::DataEnter>;
+using ExitDataOpLowering =
+ ACCDataCallLowering<ExitDataOp, RuntimeFunction::ACCRTL_tgt_acc_data_exit,
+ ACCDataCallKind::DataExit>;
+using UpdateOpLowering =
+ ACCDataCallLowering<UpdateOp, RuntimeFunction::ACCRTL_tgt_acc_data_update,
+ ACCDataCallKind::DataEnter>;
+
+/// A data clause operation only describes a mapping; the runtime calls of the
+/// construct carry everything it said, so the operation itself goes away once
+/// they are in place. Whatever else referred to the mapped object is given the
+/// address of that object, which is what the clause result stood for.
+template <typename OpTy>
+struct DataEntryOpLowering : public ConvertOpToLLVMPattern<OpTy> {
+ using ConvertOpToLLVMPattern<OpTy>::ConvertOpToLLVMPattern;
+ using OpAdaptor = typename OpTy::Adaptor;
+
+ LogicalResult
+ matchAndRewrite(OpTy op, OpAdaptor adaptor,
+ ConversionPatternRewriter &rewriter) const override {
+ rewriter.replaceOp(op, adaptor.getVar());
+ return success();
+ }
+};
+
+/// A data exit operation and the bounds of a mapping have no result to stand
+/// in for, so they are simply removed with the construct that used them.
+template <typename OpTy>
+struct EraseDataClauseOp : public ConvertOpToLLVMPattern<OpTy> {
+ using ConvertOpToLLVMPattern<OpTy>::ConvertOpToLLVMPattern;
+ using OpAdaptor = typename OpTy::Adaptor;
+
+ LogicalResult
+ matchAndRewrite(OpTy op, OpAdaptor,
+ ConversionPatternRewriter &rewriter) const override {
+ rewriter.eraseOp(op);
+ return success();
+ }
+};
+
+} // namespace
+
+void mlir::configureACCDataDirectiveConversionLegality(
+ ConversionTarget &target) {
+ // The map entries themselves stay legal: a construct removes the ones it
+ // consumes, and the ones describing a construct that is lowered elsewhere
+ // have to survive this conversion.
+ target.addIllegalOp<acc::DataOp, acc::EnterDataOp, acc::ExitDataOp,
+ acc::UpdateOp>();
+}
+
+void mlir::populateACCDataClauseOpPatterns(LLVMTypeConverter &converter,
+ RewritePatternSet &patterns) {
+ // Bounds describe a mapping rather than data, and they are removed together
+ // with the clause operations holding them, so their type only has to survive
+ // this conversion.
+ converter.addConversion(
+ [](acc::DataBoundsType type) -> Type { return type; });
+
+ patterns.add<
+ DataEntryOpLowering<acc::MapInfoOp>, DataEntryOpLowering<acc::CopyinOp>,
+ DataEntryOpLowering<acc::CreateOp>, DataEntryOpLowering<acc::PresentOp>,
+ DataEntryOpLowering<acc::NoCreateOp>, DataEntryOpLowering<acc::AttachOp>,
+ DataEntryOpLowering<acc::DevicePtrOp>,
+ DataEntryOpLowering<acc::GetDevicePtrOp>,
+ DataEntryOpLowering<acc::UpdateDeviceOp>,
+ DataEntryOpLowering<acc::PrivateOp>,
+ DataEntryOpLowering<acc::FirstprivateOp>,
+ DataEntryOpLowering<acc::FirstprivateMapInitialOp>,
+ DataEntryOpLowering<acc::DeclareDeviceResidentOp>,
+ EraseDataClauseOp<acc::CopyoutOp>, EraseDataClauseOp<acc::DeleteOp>,
+ EraseDataClauseOp<acc::DetachOp>, EraseDataClauseOp<acc::UpdateHostOp>,
+ EraseDataClauseOp<acc::DataBoundsOp>>(converter);
+}
+
+void mlir::populateACCDataDirectivePatterns(
+ LLVMTypeConverter &converter, RewritePatternSet &patterns,
+ acc::OpenACCSupport &accSupport, Region &globalSymbolRegion,
+ SymbolTable &symbolTable, const acc::ACCRuntimeCallConfig &config,
+ acc::DeviceType clauseDeviceType) {
+ patterns.add<DataOpLowering, EnterDataOpLowering, ExitDataOpLowering,
+ UpdateOpLowering>(converter, accSupport, globalSymbolRegion,
+ symbolTable, config, clauseDeviceType);
+}
diff --git a/mlir/lib/Conversion/OpenACCToLLVM/ACCDataRuntime.cpp b/mlir/lib/Conversion/OpenACCToLLVM/ACCDataRuntime.cpp
new file mode 100644
index 0000000000000..6eb4547bf9491
--- /dev/null
+++ b/mlir/lib/Conversion/OpenACCToLLVM/ACCDataRuntime.cpp
@@ -0,0 +1,477 @@
+//===- ACCDataRuntime.cpp - Emit OpenACC data runtime arguments -*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Builds the mapping argument arrays used by OpenACC data runtime entry points.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Conversion/OpenACCToLLVM/ACCDataRuntime.h"
+#include "mlir/Conversion/OpenACCToLLVM/ACCToLLVMUtils.h"
+
+#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsType.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/Support/Debug.h"
+
+#include <algorithm>
+
+#define DEBUG_TYPE "acc-data-runtime"
+
+using namespace mlir;
+using namespace mlir::acc;
+
+static Value remap(Value value, ConversionPatternRewriter &rewriter) {
+ if (!value)
+ return {};
+ if (Value converted = rewriter.getRemappedValue(value))
+ return converted;
+ return value;
+}
+
+static Value constantI64(Location loc, int64_t value,
+ ConversionPatternRewriter &rewriter) {
+ return LLVM::ConstantOp::create(rewriter, loc, rewriter.getI64Type(), value);
+}
+
+static Value getPointer(Location loc, Value value,
+ ConversionPatternRewriter &rewriter) {
+ Type ptrTy = LLVM::LLVMPointerType::get(rewriter.getContext());
+ if (!value)
+ return LLVM::ZeroOp::create(rewriter, loc, ptrTy);
+ if (value.getType() == ptrTy)
+ return value;
+
+ if (isa<PointerLikeType>(value.getType()))
+ return castPointerLikeTypeIfNeeded(rewriter, loc, value, ptrTy);
+
+ Value one = LLVM::ConstantOp::create(rewriter, loc, rewriter.getI32Type(), 1);
+ Value storage =
+ LLVM::AllocaOp::create(rewriter, loc, ptrTy, value.getType(), one);
+ LLVM::StoreOp::create(rewriter, loc, value, storage);
+ return storage;
+}
+
+/// Returns the address of a slot holding \p value, so that the runtime reads
+/// the value from memory rather than receiving it directly.
+static Value getIndirectPointer(Location loc, Value value,
+ ConversionPatternRewriter &rewriter) {
+ Type ptrTy = LLVM::LLVMPointerType::get(rewriter.getContext());
+ Value one = LLVM::ConstantOp::create(rewriter, loc, rewriter.getI32Type(), 1);
+ Value storage =
+ LLVM::AllocaOp::create(rewriter, loc, ptrTy, value.getType(), one);
+ LLVM::StoreOp::create(rewriter, loc, value, storage);
+ return storage;
+}
+
+Value mlir::createACCDataArray(Location loc, Type elementType,
+ ArrayRef<Value> values, RewriterBase &rewriter) {
+ Type ptrTy = LLVM::LLVMPointerType::get(rewriter.getContext());
+ Type i32Ty = rewriter.getI32Type();
+ Value count = LLVM::ConstantOp::create(rewriter, loc, i32Ty, values.size());
+ Value array =
+ LLVM::AllocaOp::create(rewriter, loc, ptrTy, elementType, count);
+ for (auto [index, value] : llvm::enumerate(values)) {
+ Value indexValue = LLVM::ConstantOp::create(rewriter, loc, i32Ty, index);
+ Value element = LLVM::GEPOp::create(rewriter, loc, ptrTy, elementType,
+ array, ValueRange{indexValue});
+ LLVM::StoreOp::create(rewriter, loc, value, element);
+ }
+ return array;
+}
+
+/// The address of the mapped object, as the runtime expects to receive it.
+/// Privatized storage only ever exists on the device, so there is no host
+/// address to give. A device address names storage the runtime must not look
+/// up, and an object mapped by value has no address at all, so both are handed
+/// over in a slot the runtime reads them from.
+static Value getMappedObjectPointer(Location loc, Value operand,
+ Value converted, MapFlags mapFlags,
+ ConversionPatternRewriter &rewriter) {
+ Type ptrTy = LLVM::LLVMPointerType::get(rewriter.getContext());
+ if (isa<PrivateType>(operand.getType()))
+ return LLVM::ZeroOp::create(rewriter, loc, ptrTy);
+ if (converted && converted.getType() == ptrTy &&
+ bitEnumContainsAny(mapFlags, MapFlags::devptr))
+ return getIndirectPointer(loc, converted, rewriter);
+ return getPointer(loc, converted, rewriter);
+}
+
+static std::optional<MapFlags> computePackedMapFlags(Operation *mapOp) {
+ if (auto mapInfo = dyn_cast<MapInfoOp>(mapOp))
+ return mapInfo.getMapFlags();
+ if (isa<ACC_DATA_ENTRY_OPS>(mapOp))
+ return computeDataClauseMapFlags(mapOp, hasAttachPoint(mapOp));
+ return std::nullopt;
+}
+
+static Value getMapSize(Operation *mapOp, Value converted, MapFlags mapFlags,
+ OpenACCSupport &accSupport,
+ ConversionPatternRewriter &rewriter) {
+ Location loc = mapOp->getLoc();
+ ModuleOp module = mapOp->getParentOfType<ModuleOp>();
+ // An object mapped by value is passed as the argument itself, so its size is
+ // the size of what is passed and not of the object it was read from. The two
+ // differ when the argument holds one field of a decomposed aggregate.
+ if (bitEnumContainsAny(mapFlags, MapFlags::literal) && converted) {
+ if (auto sizeAndAlignment = acc::getTypeSizeAndAlignment(
+ converted.getType(), module, &accSupport))
+ return constantI64(loc, sizeAndAlignment->first.getFixedValue(),
+ rewriter);
+ }
+ if (Value size = acc::getMapSize(mapOp))
+ return castToI64(loc, remap(size, rewriter), rewriter);
+
+ Value var = acc::getVar(mapOp);
+ if (!var)
+ return constantI64(loc, 0, rewriter);
+ Type varType = acc::getVarType(mapOp);
+ if (!varType)
+ varType = var.getType();
+ // A memref is handed over with a descriptor stating its extents and element
+ // size, which is what states the size of such a mapping.
+ if (isa<MemRefType>(var.getType()) || isa<MemRefType>(varType))
+ return constantI64(loc, 0, rewriter);
+
+ // A data clause operation that was not turned into an `acc.map_info` has no
+ // size of its own, so it is sized the way that operation would have been.
+ // A size that is not statically known is left to the runtime, as an unsized
+ // mapping is.
+ int64_t size =
+ acc::computeMapInfoSizeBytes(var, varType, acc::getDataDescKind(mapOp),
+ acc::getBounds(mapOp), &accSupport);
+ return constantI64(loc, std::max<int64_t>(size, 0), rewriter);
+}
+
+static Value getElementSize(Operation *mapOp, OpenACCSupport &accSupport,
+ ConversionPatternRewriter &rewriter) {
+ Location loc = mapOp->getLoc();
+ ModuleOp module = mapOp->getParentOfType<ModuleOp>();
+ if (std::optional<int64_t> size = acc::getMapElementSize(mapOp))
+ return constantI64(loc, *size, rewriter);
+
+ Type type = acc::getVarType(mapOp);
+ if (auto shaped = dyn_cast<ShapedType>(type))
+ type = shaped.getElementType();
+ if (!type)
+ return constantI64(loc, 0, rewriter);
+ if (auto sizeAndAlignment =
+ acc::getTypeSizeAndAlignment(type, module, &accSupport))
+ return constantI64(loc, sizeAndAlignment->first.getFixedValue(), rewriter);
+ return constantI64(loc, 0, rewriter);
+}
+
+static Value getBoundValue(Value value, Location loc,
+ ConversionPatternRewriter &rewriter) {
+ if (!value)
+ return constantI64(loc, 0, rewriter);
+ return castToI64(loc, remap(value, rewriter), rewriter);
+}
+
+Value mlir::createACCDataDescriptor(Location loc, Value baseDescriptor,
+ Type baseDescriptorType, uint32_t version,
+ ValueRange bounds, Value elementSize,
+ ConversionPatternRewriter &rewriter) {
+ MLIRContext *context = rewriter.getContext();
+ Type i8Ty = rewriter.getI8Type();
+ Type i32Ty = rewriter.getI32Type();
+ Type i64Ty = rewriter.getI64Type();
+ Type ptrTy = LLVM::LLVMPointerType::get(context);
+
+ Value versionValue = LLVM::ConstantOp::create(
+ rewriter, loc, i32Ty,
+ version | static_cast<uint32_t>(
+ getDataDescriptorKind(DataDescriptor::AccDataDescOpenACC)));
+ baseDescriptor = LLVM::InsertValueOp::create(
+ rewriter, loc, baseDescriptorType, baseDescriptor, versionValue,
+ ArrayRef<int64_t>{0});
+
+ Type descriptorType = getDataDescriptorType(
+ context, DataDescriptor::AccDataDescOpenACC, baseDescriptorType);
+ Value descriptor = LLVM::ZeroOp::create(rewriter, loc, descriptorType);
+ Value rank = LLVM::ConstantOp::create(rewriter, loc, i8Ty, bounds.size());
+
+ SmallVector<Value> lowerBounds;
+ SmallVector<Value> upperBounds;
+ SmallVector<Value> extents;
+ SmallVector<Value> strides;
+ SmallVector<Value> starts;
+ for (Value boundValue : bounds) {
+ auto bound = cast<DataBoundsOp>(boundValue.getDefiningOp());
+ lowerBounds.push_back(
+ getBoundValue(bound.getLowerbound(), bound.getLoc(), rewriter));
+ upperBounds.push_back(
+ getBoundValue(bound.getUpperbound(), bound.getLoc(), rewriter));
+ Value sourceExtent =
+ bound.getSourceExtent() ? bound.getSourceExtent() : bound.getExtent();
+ extents.push_back(getBoundValue(sourceExtent, bound.getLoc(), rewriter));
+ Value stride = getBoundValue(bound.getStride(), bound.getLoc(), rewriter);
+ if (!bound.getStrideInBytes())
+ stride =
+ LLVM::MulOp::create(rewriter, bound.getLoc(), stride, elementSize);
+ strides.push_back(stride);
+ starts.push_back(
+ getBoundValue(bound.getStartIdx(), bound.getLoc(), rewriter));
+ }
+
+ Value lowerBoundsArray =
+ createACCDataArray(loc, i64Ty, lowerBounds, rewriter);
+ Value upperBoundsArray =
+ createACCDataArray(loc, i64Ty, upperBounds, rewriter);
+ Value extentsArray = createACCDataArray(loc, i64Ty, extents, rewriter);
+ Value stridesArray = createACCDataArray(loc, i64Ty, strides, rewriter);
+ Value startsArray = createACCDataArray(loc, i64Ty, starts, rewriter);
+
+ using Field = AccDataDescOpenACCField;
+ auto insert = [&](Value value, Field field) {
+ descriptor = LLVM::InsertValueOp::create(
+ rewriter, loc, descriptorType, descriptor, value,
+ ArrayRef<int64_t>{getDataDescriptorFieldIndex(field)});
+ };
+ insert(baseDescriptor, Field::Base);
+ insert(rank, Field::Rank);
+ insert(elementSize, Field::ElementSize);
+ insert(lowerBoundsArray, Field::LowerBounds);
+ insert(upperBoundsArray, Field::UpperBounds);
+ insert(extentsArray, Field::Extents);
+ insert(stridesArray, Field::StridesInBytes);
+ insert(startsArray, Field::StartIndices);
+
+ Value one = LLVM::ConstantOp::create(rewriter, loc, i32Ty, 1);
+ Value storage =
+ LLVM::AllocaOp::create(rewriter, loc, ptrTy, descriptorType, one);
+ LLVM::StoreOp::create(rewriter, loc, descriptor, storage);
+ return storage;
+}
+
+Value mlir::createACCMemRefDescriptorWrapperArg(
+ Location loc, MemRefType memrefType, Value convertedMemref,
+ ValueRange bounds, Value elementSize, ConversionPatternRewriter &rewriter) {
+ MLIRContext *context = rewriter.getContext();
+ Type i8Ty = rewriter.getI8Type();
+ Type i32Ty = rewriter.getI32Type();
+ Type ptrTy = LLVM::LLVMPointerType::get(context);
+ using Field = AccDataDescMemRefField;
+ Type descriptorType =
+ getDataDescriptorType(context, DataDescriptor::AccDataDescMemRef);
+ auto version = static_cast<uint32_t>(
+ getDataDescriptorKind(DataDescriptor::AccDataDescMemRef));
+ Value descriptor = LLVM::ZeroOp::create(rewriter, loc, descriptorType);
+ Value rank =
+ LLVM::ConstantOp::create(rewriter, loc, i8Ty, memrefType.getRank());
+ auto insert = [&](Value value, Field field) {
+ descriptor = LLVM::InsertValueOp::create(
+ rewriter, loc, descriptorType, descriptor, value,
+ ArrayRef<int64_t>{getDataDescriptorFieldIndex(field)});
+ };
+ insert(rank, Field::Rank);
+ insert(elementSize, Field::ElementSize);
+ insert(getPointer(loc, convertedMemref, rewriter), Field::MemRefDescriptor);
+ if (!bounds.empty())
+ return createACCDataDescriptor(loc, descriptor, descriptorType, version,
+ bounds, elementSize, rewriter);
+
+ insert(LLVM::ConstantOp::create(rewriter, loc, i32Ty, version),
+ Field::Version);
+ Value one = LLVM::ConstantOp::create(rewriter, loc, i32Ty, 1);
+ Value storage =
+ LLVM::AllocaOp::create(rewriter, loc, ptrTy, descriptorType, one);
+ LLVM::StoreOp::create(rewriter, loc, descriptor, storage);
+ return storage;
+}
+
+Value mlir::createACCArgumentDescriptor(Operation *mapOp,
+ Value convertedOperand,
+ OpenACCSupport &accSupport,
+ ConversionPatternRewriter &rewriter) {
+ Location loc = mapOp->getLoc();
+ MLIRContext *context = rewriter.getContext();
+ Type i32Ty = rewriter.getI32Type();
+ Type ptrTy = LLVM::LLVMPointerType::get(context);
+ SmallVector<Value> bounds = acc::getBounds(mapOp);
+ DataDescKind descKind = acc::getDataDescKind(mapOp);
+ // Only a descriptor stating bounds reads the element size, so it is
+ // materialized where it is used, leaving no constant behind for a mapping
+ // that needs no descriptor at all.
+ auto elementSize = [&] {
+ return getElementSize(mapOp, accSupport, rewriter);
+ };
+
+ Value var = acc::getVar(mapOp);
+ if (var && isa<MemRefType>(var.getType())) {
+ auto memrefType = cast<MemRefType>(var.getType());
+ return createACCMemRefDescriptorWrapperArg(
+ loc, memrefType, convertedOperand, bounds, elementSize(), rewriter);
+ }
+
+ if (bitEnumContainsAny(descKind, DataDescKind::cfi)) {
+ using Field = AccDataDescCFIField;
+ Type descriptorType =
+ getDataDescriptorType(context, DataDescriptor::AccDataDescCFI);
+ auto version = static_cast<uint32_t>(
+ getDataDescriptorKind(DataDescriptor::AccDataDescCFI));
+ Value descriptor = LLVM::ZeroOp::create(rewriter, loc, descriptorType);
+ Value descriptorStorage = remap(acc::getDesc(mapOp), rewriter);
+ if (!descriptorStorage)
+ descriptorStorage = convertedOperand;
+ descriptor = LLVM::InsertValueOp::create(
+ rewriter, loc, descriptorType, descriptor,
+ getPointer(loc, descriptorStorage, rewriter),
+ ArrayRef<int64_t>{getDataDescriptorFieldIndex(Field::CFIDescriptor)});
+ if (!bounds.empty())
+ return createACCDataDescriptor(loc, descriptor, descriptorType, version,
+ bounds, elementSize(), rewriter);
+
+ Value versionValue =
+ LLVM::ConstantOp::create(rewriter, loc, i32Ty, version);
+ descriptor = LLVM::InsertValueOp::create(
+ rewriter, loc, descriptorType, descriptor, versionValue,
+ ArrayRef<int64_t>{getDataDescriptorFieldIndex(Field::Version)});
+ Value one = LLVM::ConstantOp::create(rewriter, loc, i32Ty, 1);
+ Value storage =
+ LLVM::AllocaOp::create(rewriter, loc, ptrTy, descriptorType, one);
+ LLVM::StoreOp::create(rewriter, loc, descriptor, storage);
+ return storage;
+ }
+
+ if (!bounds.empty()) {
+ Type descriptorType =
+ getDataDescriptorType(context, DataDescriptor::AccDataDescGeneric);
+ auto version = static_cast<uint32_t>(
+ getDataDescriptorKind(DataDescriptor::AccDataDescGeneric));
+ Value descriptor = LLVM::ZeroOp::create(rewriter, loc, descriptorType);
+ return createACCDataDescriptor(loc, descriptor, descriptorType, version,
+ bounds, elementSize(), rewriter);
+ }
+ return LLVM::ZeroOp::create(rewriter, loc, ptrTy);
+}
+
+SmallVector<Value> mlir::ACCDataRuntimeArgs::getCallArgs() const {
+ return {ident, flags, deviceType, argNum, argBasePtrs, argPtrs,
+ argSizes, argTypes, argNames, argMappers, argDescs};
+}
+
+LogicalResult mlir::emitACCDataRuntimeArgs(
+ Location loc, ValueRange mappingOperands, ValueRange convertedOperands,
+ ConversionPatternRewriter &rewriter, Region &globalSymbolRegion,
+ acc::OpenACCSupport &accSupport, const acc::ACCRuntimeCallConfig &config,
+ ACCDataRuntimeArgs &runtimeArgs, ACCDataCallKind callKind,
+ SymbolTable *symbolTable) {
+ if (mappingOperands.size() != convertedOperands.size())
+ return failure();
+
+ Type i32Ty = rewriter.getI32Type();
+ Type i64Ty = rewriter.getI64Type();
+ Type ptrTy = LLVM::LLVMPointerType::get(rewriter.getContext());
+
+ runtimeArgs.ident =
+ createIdent(loc, getParentFunctionName(mappingOperands), rewriter,
+ globalSymbolRegion, config, symbolTable);
+ runtimeArgs.flags = constantI64(loc, 0, rewriter);
+ runtimeArgs.deviceType = constantI64(
+ loc, config.getDeviceTypeRuntimeValue(DeviceType::None), rewriter);
+ runtimeArgs.argMappers = LLVM::ZeroOp::create(rewriter, loc, ptrTy);
+
+ // An aggregate passed by value is decomposed into one argument per field by
+ // the target ABI, so the runtime is told about one object per field as well.
+ // Each of them keeps the map entry of the aggregate it came from.
+ SmallVector<std::pair<Value, Value>> mappedObjects;
+ for (auto [operand, converted] :
+ llvm::zip_equal(mappingOperands, convertedOperands)) {
+ auto structType = dyn_cast<LLVM::LLVMStructType>(converted.getType());
+ if (structType && !structType.isIdentified() &&
+ structType.getBody().size() > 1) {
+ for (unsigned field = 0, fields = structType.getBody().size();
+ field != fields; ++field)
+ mappedObjects.emplace_back(
+ operand,
+ LLVM::ExtractValueOp::create(rewriter, loc, converted, field));
+ continue;
+ }
+ mappedObjects.emplace_back(operand, converted);
+ }
+
+ runtimeArgs.argNum =
+ LLVM::ConstantOp::create(rewriter, loc, i32Ty, mappedObjects.size());
+
+ SmallVector<Value> bases;
+ SmallVector<Value> pointers;
+ SmallVector<Value> sizes;
+ SmallVector<Value> types;
+ SmallVector<Value> names;
+ SmallVector<Value> descriptors;
+ for (auto [operand, converted] : mappedObjects) {
+ Operation *mapOp = operand.getDefiningOp();
+ if (!mapOp || !isa<MapInfoOp, ACC_DATA_ENTRY_OPS>(mapOp))
+ return failure();
+ std::optional<MapFlags> mapFlags = computePackedMapFlags(mapOp);
+ if (!mapFlags)
+ return failure();
+ *mapFlags = config.postProcessMapFlags(mapOp, *mapFlags);
+ LLVM_DEBUG(llvm::dbgs() << "mapping " << *mapOp << "\n as "
+ << config.formatMapFlags(*mapFlags) << "\n");
+
+ Location mapLoc = mapOp->getLoc();
+ pointers.push_back(getMappedObjectPointer(mapLoc, operand, converted,
+ *mapFlags, rewriter));
+
+ Value base;
+ if (Value attach = acc::getVarPtrPtr(mapOp))
+ base = remap(attach, rewriter);
+ else if (bitEnumContainsAny(*mapFlags, MapFlags::ptr_and_obj))
+ base = remap(acc::getDesc(mapOp), rewriter);
+ // An object that only lives on the device has no host address to state as
+ // its base.
+ if (bitEnumContainsAny(*mapFlags, MapFlags::device_resident))
+ base = {};
+ bases.push_back(getPointer(mapLoc, base, rewriter));
+
+ // A local object that lives on the device is torn down when the region
+ // that declared it ends, which the runtime only does for a mapping that
+ // states it is deleted.
+ if (callKind == ACCDataCallKind::DataExit &&
+ bitEnumContainsAny(*mapFlags, MapFlags::device_resident)) {
+ Value var = acc::getVar(mapOp);
+ if (var &&
+ !isa_and_nonnull<AddressOfGlobalOpInterface>(var.getDefiningOp()))
+ *mapFlags = *mapFlags | MapFlags::delete_;
+ }
+
+ sizes.push_back(
+ getMapSize(mapOp, converted, *mapFlags, accSupport, rewriter));
+ types.push_back(constantI64(
+ mapLoc, config.getMapFlagsRuntimeValue(*mapFlags), rewriter));
+
+ // The runtime resolves an object that lives on the device against the
+ // symbols of the binary by this name, so it is the name the object is
+ // emitted under rather than the one the source spells.
+ VariableNameConfig nameConfig;
+ nameConfig.preferDemangledName = false;
+ std::string name = accSupport.getVariableName(operand, nameConfig);
+ if (name.empty()) {
+ names.push_back(LLVM::ZeroOp::create(rewriter, mapLoc, ptrTy));
+ } else {
+ names.push_back(getOrCreateGlobalString(
+ mapLoc, rewriter, getInternalGlobalName("var_name", name), name,
+ globalSymbolRegion, symbolTable));
+ }
+ descriptors.push_back(
+ createACCArgumentDescriptor(mapOp, converted, accSupport, rewriter));
+ }
+
+ runtimeArgs.argBasePtrs = createACCDataArray(loc, ptrTy, bases, rewriter);
+ runtimeArgs.argPtrs = createACCDataArray(loc, ptrTy, pointers, rewriter);
+ runtimeArgs.argSizes = createACCDataArray(loc, i64Ty, sizes, rewriter);
+ runtimeArgs.argTypes = createACCDataArray(loc, i64Ty, types, rewriter);
+ runtimeArgs.argNames = createACCDataArray(loc, ptrTy, names, rewriter);
+ runtimeArgs.argDescs = createACCDataArray(loc, ptrTy, descriptors, rewriter);
+ return success();
+}
diff --git a/mlir/lib/Conversion/OpenACCToLLVM/ACCExecutableDirectivePatterns.cpp b/mlir/lib/Conversion/OpenACCToLLVM/ACCExecutableDirectivePatterns.cpp
index ae4090d00bd26..2d36d4b66c32f 100644
--- a/mlir/lib/Conversion/OpenACCToLLVM/ACCExecutableDirectivePatterns.cpp
+++ b/mlir/lib/Conversion/OpenACCToLLVM/ACCExecutableDirectivePatterns.cpp
@@ -15,83 +15,30 @@
#include "mlir/Conversion/OpenACCToLLVM/ACCToLLVMUtils.h"
#include "mlir/Conversion/LLVMCommon/Pattern.h"
-#include "mlir/Dialect/Arith/IR/Arith.h"
#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
#include "mlir/Dialect/OpenACC/OpenACC.h"
#include "mlir/IR/PatternMatch.h"
#include "llvm/ADT/STLExtras.h"
-#include "llvm/ADT/STLFunctionalExtras.h"
#include <cstdint>
-#include <iterator>
using namespace mlir;
using namespace mlir::acc;
namespace {
-static Value castToI64(Location loc, Value value,
- ConversionPatternRewriter &rewriter) {
- Type i64Ty = IntegerType::get(rewriter.getContext(), 64);
- unsigned bitwidth = value.getType().getIntOrFloatBitWidth();
- if (bitwidth > 64)
- return arith::TruncIOp::create(rewriter, loc, i64Ty, value);
- if (bitwidth < 64)
- return arith::ExtSIOp::create(rewriter, loc, i64Ty, value);
- return value;
-}
-
-static Value getAsyncQueue(WaitOp op, ConversionPatternRewriter &rewriter,
- const ACCRuntimeCallConfig &config) {
- Location loc = op->getLoc();
- Type i64Ty = IntegerType::get(rewriter.getContext(), 64);
- if (op.getAsync())
- return LLVM::ConstantOp::create(rewriter, loc, i64Ty,
- config.getAsyncNoValueRuntimeValue());
- if (Value asyncValue = op.getAsyncOperand()) {
- asyncValue = rewriter.getRemappedValue(asyncValue);
- return castToI64(loc, asyncValue, rewriter);
- }
- return LLVM::ConstantOp::create(rewriter, loc, i64Ty,
- config.getAsyncSyncRuntimeValue());
-}
-
-static LogicalResult createIfThen(Location loc, Value ifCond,
- ConversionPatternRewriter &rewriter,
- function_ref<LogicalResult()> thenFn) {
- Block *parentBlock = rewriter.getInsertionBlock();
- Block *continueBlock =
- rewriter.splitBlock(parentBlock, rewriter.getInsertionPoint());
- Block *thenBlock = rewriter.createBlock(
- parentBlock->getParent(), std::next(Region::iterator(parentBlock)));
-
- rewriter.setInsertionPointToEnd(parentBlock);
- LLVM::CondBrOp::create(rewriter, loc, ifCond, thenBlock, ValueRange{},
- continueBlock, ValueRange{});
-
- rewriter.setInsertionPointToStart(thenBlock);
- LogicalResult result = thenFn();
- rewriter.setInsertionPointToEnd(thenBlock);
- LLVM::BrOp::create(rewriter, loc, ValueRange{}, continueBlock);
- rewriter.setInsertionPointToStart(continueBlock);
- return result;
-}
-
-/// Run \p emitFn, guarded by a branch on \p ifCond when it is present.
-static LogicalResult emitGuardedByIfCond(Location loc, Value ifCond,
- ConversionPatternRewriter &rewriter,
- function_ref<LogicalResult()> emitFn) {
- if (ifCond)
- return createIfThen(loc, ifCond, rewriter, emitFn);
- return emitFn();
-}
-
template <typename OpTy>
struct ACCExecutableDirectivePattern : public ConvertOpToLLVMPattern<OpTy> {
ACCExecutableDirectivePattern(const LLVMTypeConverter &converter,
+ Region &globalSymbolRegion,
+ SymbolTable &symbolTable,
const ACCRuntimeCallConfig &config,
PatternBenefit benefit = 1)
- : ConvertOpToLLVMPattern<OpTy>(converter, benefit), config(config) {}
+ : ConvertOpToLLVMPattern<OpTy>(converter, benefit),
+ globalSymbolRegion(globalSymbolRegion), symbolTable(symbolTable),
+ config(config) {}
+ Region &globalSymbolRegion;
+ SymbolTable &symbolTable;
ACCRuntimeCallConfig config;
};
@@ -102,47 +49,18 @@ struct WaitOpLowering : public ACCExecutableDirectivePattern<WaitOp> {
matchAndRewrite(WaitOp op, WaitOp::Adaptor,
ConversionPatternRewriter &rewriter) const override {
Location loc = op->getLoc();
- ModuleOp module = op->getParentOfType<ModuleOp>();
- Type i32Ty = rewriter.getI32Type();
- Type i64Ty = rewriter.getI64Type();
- Type ptrTy = LLVM::LLVMPointerType::get(rewriter.getContext());
auto emitWait = [&]() -> LogicalResult {
- Value asyncQueue = getAsyncQueue(op, rewriter, config);
+ Value asyncOperand = op.getAsyncOperand()
+ ? rewriter.getRemappedValue(op.getAsyncOperand())
+ : Value();
+ Value asyncQueue =
+ getAsyncQueue(loc, asyncOperand, op.getAsync(), rewriter, config);
SmallVector<Value> waitValues;
for (Value operand : op.getWaitOperands())
- waitValues.push_back(
- castToI64(loc, rewriter.getRemappedValue(operand), rewriter));
-
- unsigned size = waitValues.size();
- Value waitNum = LLVM::ConstantOp::create(rewriter, loc, i32Ty, size);
- Value waitList;
- if (size == 0) {
- waitList = LLVM::ZeroOp::create(rewriter, loc, ptrTy);
- } else {
- waitList = LLVM::AllocaOp::create(rewriter, loc, ptrTy, i64Ty, waitNum);
- for (auto [index, waitValue] : llvm::enumerate(waitValues)) {
- Value idx = LLVM::ConstantOp::create(rewriter, loc, i32Ty,
- static_cast<int64_t>(index));
- Value elementPtr = LLVM::GEPOp::create(
- rewriter, loc, ptrTy, i64Ty, waitList, ArrayRef<Value>{idx});
- LLVM::StoreOp::create(rewriter, loc, waitValue, elementPtr);
- }
- }
-
- StringRef functionName = getParentFunctionName(waitValues);
- if (functionName.empty())
- functionName = getParentFunctionName(op);
- Value ident = createIdent(loc, functionName, rewriter, module, config);
- Value flags = LLVM::ConstantOp::create(rewriter, loc, i64Ty, 0);
- Value deviceType = LLVM::ConstantOp::create(
- rewriter, loc, i64Ty,
- config.getDeviceTypeRuntimeValue(DeviceType::None));
- Value deviceNum = LLVM::ConstantOp::create(rewriter, loc, i32Ty, 0);
-
- return createRuntimeCall(
- loc, rewriter, module, RuntimeFunction::ACCRTL_tgt_acc_wait, config,
- {ident, flags, deviceType, deviceNum, waitNum, waitList, asyncQueue});
+ waitValues.push_back(rewriter.getRemappedValue(operand));
+ return emitWaitCall(loc, waitValues, asyncQueue, rewriter,
+ globalSymbolRegion, symbolTable, config);
};
if (failed(emitGuardedByIfCond(loc, op.getIfCond(), rewriter, emitWait)))
@@ -159,17 +77,20 @@ struct WaitOpLowering : public ACCExecutableDirectivePattern<WaitOp> {
static LogicalResult
emitDeviceOperationCall(Location loc, RuntimeFunction fn, DeviceType deviceType,
Value deviceNum, StringRef functionName,
- ModuleOp module, ConversionPatternRewriter &rewriter,
+ Region &globalSymbolRegion, SymbolTable &symbolTable,
+ ConversionPatternRewriter &rewriter,
const ACCRuntimeCallConfig &config) {
Type i64Ty = rewriter.getI64Type();
Value deviceTypeValue = LLVM::ConstantOp::create(
rewriter, loc, i64Ty, config.getDeviceTypeRuntimeValue(deviceType));
- Value ident = createIdent(loc, functionName, rewriter, module, config);
+ Value ident = createIdent(loc, functionName, rewriter, globalSymbolRegion,
+ config, &symbolTable);
Value flags = LLVM::ConstantOp::create(rewriter, loc, i64Ty, 0);
Value deviceNumValue =
deviceNum ? castToI64(loc, deviceNum, rewriter)
: LLVM::ConstantOp::create(rewriter, loc, i64Ty, -1);
- return createRuntimeCall(loc, rewriter, module, fn, config,
+ return createRuntimeCall(loc, rewriter, globalSymbolRegion, symbolTable, fn,
+ config,
{ident, flags, deviceTypeValue, deviceNumValue});
}
@@ -177,8 +98,9 @@ static LogicalResult rewriteInitOrShutdown(Operation *op, Value deviceNum,
ArrayAttr deviceTypesAttr,
Value ifCond, bool isInit,
ConversionPatternRewriter &rewriter,
+ Region &globalSymbolRegion,
+ SymbolTable &symbolTable,
const ACCRuntimeCallConfig &config) {
- ModuleOp module = op->getParentOfType<ModuleOp>();
Location loc = op->getLoc();
auto emitCalls = [&]() -> LogicalResult {
@@ -189,7 +111,8 @@ static LogicalResult rewriteInitOrShutdown(Operation *op, Value deviceNum,
auto emitOne = [&](DeviceType deviceType) {
return emitDeviceOperationCall(loc, fn, deviceType, deviceNum,
- functionName, module, rewriter, config);
+ functionName, globalSymbolRegion,
+ symbolTable, rewriter, config);
};
if (!deviceTypesAttr)
@@ -216,9 +139,9 @@ struct InitOpLowering : public ACCExecutableDirectivePattern<InitOp> {
LogicalResult
matchAndRewrite(InitOp op, InitOp::Adaptor adaptor,
ConversionPatternRewriter &rewriter) const override {
- return rewriteInitOrShutdown(op, adaptor.getDeviceNum(),
- op.getDeviceTypesAttr(), op.getIfCond(),
- /*isInit=*/true, rewriter, config);
+ return rewriteInitOrShutdown(
+ op, adaptor.getDeviceNum(), op.getDeviceTypesAttr(), op.getIfCond(),
+ /*isInit=*/true, rewriter, globalSymbolRegion, symbolTable, config);
}
};
@@ -229,9 +152,9 @@ struct ShutdownOpLowering : public ACCExecutableDirectivePattern<ShutdownOp> {
LogicalResult
matchAndRewrite(ShutdownOp op, ShutdownOp::Adaptor adaptor,
ConversionPatternRewriter &rewriter) const override {
- return rewriteInitOrShutdown(op, adaptor.getDeviceNum(),
- op.getDeviceTypesAttr(), op.getIfCond(),
- /*isInit=*/false, rewriter, config);
+ return rewriteInitOrShutdown(
+ op, adaptor.getDeviceNum(), op.getDeviceTypesAttr(), op.getIfCond(),
+ /*isInit=*/false, rewriter, globalSymbolRegion, symbolTable, config);
}
};
@@ -241,17 +164,17 @@ struct SetOpLowering : public ACCExecutableDirectivePattern<SetOp> {
LogicalResult
matchAndRewrite(SetOp op, SetOp::Adaptor adaptor,
ConversionPatternRewriter &rewriter) const override {
- ModuleOp module = op->getParentOfType<ModuleOp>();
Location loc = op.getLoc();
Type i64Ty = rewriter.getI64Type();
auto emitSet = [&]() -> LogicalResult {
if (Value asyncValue = adaptor.getDefaultAsync()) {
asyncValue = castToI64(loc, asyncValue, rewriter);
- Value ident = createIdent(loc, getParentFunctionName(asyncValue),
- rewriter, module, config);
+ Value ident =
+ createIdent(loc, getParentFunctionName(asyncValue), rewriter,
+ globalSymbolRegion, config, &symbolTable);
if (failed(createRuntimeCall(
- loc, rewriter, module,
+ loc, rewriter, globalSymbolRegion, symbolTable,
RuntimeFunction::ACCRTL_tgt_acc_set_default_async, config,
{ident, asyncValue})))
return failure();
@@ -264,17 +187,18 @@ struct SetOpLowering : public ACCExecutableDirectivePattern<SetOp> {
deviceType = deviceTypeAttr.getValue();
return emitDeviceOperationCall(
loc, RuntimeFunction::ACCRTL_tgt_acc_set_device_num, deviceType,
- deviceNum, getParentFunctionName(deviceNum), module, rewriter,
- config);
+ deviceNum, getParentFunctionName(deviceNum), globalSymbolRegion,
+ symbolTable, rewriter, config);
}
if (auto deviceTypeAttr = op.getDeviceTypeAttr()) {
Value deviceTypeValue = LLVM::ConstantOp::create(
rewriter, loc, i64Ty,
config.getDeviceTypeRuntimeValue(deviceTypeAttr.getValue()));
- Value ident = createIdent(loc, StringRef(), rewriter, module, config);
+ Value ident = createIdent(loc, StringRef(), rewriter,
+ globalSymbolRegion, config, &symbolTable);
Value flags = LLVM::ConstantOp::create(rewriter, loc, i64Ty, 0);
return createRuntimeCall(
- loc, rewriter, module,
+ loc, rewriter, globalSymbolRegion, symbolTable,
RuntimeFunction::ACCRTL_tgt_acc_set_device_type, config,
{ident, flags, deviceTypeValue});
}
@@ -298,8 +222,9 @@ void mlir::configureACCExecutableDirectiveConversionLegality(
void mlir::populateACCExecutableDirectivePatterns(
LLVMTypeConverter &converter, RewritePatternSet &patterns,
+ Region &globalSymbolRegion, SymbolTable &symbolTable,
const acc::ACCRuntimeCallConfig &config) {
patterns
.add<WaitOpLowering, InitOpLowering, ShutdownOpLowering, SetOpLowering>(
- converter, config);
+ converter, globalSymbolRegion, symbolTable, config);
}
diff --git a/mlir/lib/Conversion/OpenACCToLLVM/ACCToLLVM.cpp b/mlir/lib/Conversion/OpenACCToLLVM/ACCToLLVM.cpp
index 6a2cb4ddd7bc1..a5220afc43214 100644
--- a/mlir/lib/Conversion/OpenACCToLLVM/ACCToLLVM.cpp
+++ b/mlir/lib/Conversion/OpenACCToLLVM/ACCToLLVM.cpp
@@ -36,6 +36,7 @@ struct ConvertACCToLLVMPass
void ConvertACCToLLVMPass::runOnOperation() {
ModuleOp module = getOperation();
+ SymbolTable symbolTable(module);
LLVMTypeConverter converter(&getContext());
RewritePatternSet patterns(&getContext());
arith::populateArithToLLVMConversionPatterns(converter, patterns);
@@ -43,17 +44,21 @@ void ConvertACCToLLVMPass::runOnOperation() {
populateFuncToLLVMConversionPatterns(converter, patterns);
populateFinalizeMemRefToLLVMConversionPatterns(converter, patterns);
- // The device_type numbering is implementation-defined by the target
- // runtime. For now assume the same numbering as the OpenACC dialect.
acc::ACCRuntimeCallConfig runtimeConfig;
- acc::populateDialectIdentityDeviceTypeMapping(runtimeConfig);
- populateACCExecutableDirectivePatterns(converter, patterns, runtimeConfig);
+
+ populateACCExecutableDirectivePatterns(
+ converter, patterns, module.getBodyRegion(), symbolTable, runtimeConfig);
acc::OpenACCSupport &accSupport = getAnalysis<acc::OpenACCSupport>();
+ populateACCDataDirectivePatterns(converter, patterns, accSupport,
+ module.getBodyRegion(), symbolTable,
+ runtimeConfig);
populateACCAtomicPatterns(converter, patterns, accSupport);
+ populateACCDataClauseOpPatterns(converter, patterns);
LLVMConversionTarget target(getContext());
configureACCExecutableDirectiveConversionLegality(target);
+ configureACCDataDirectiveConversionLegality(target);
configureACCAtomicConversionLegality(target);
if (failed(applyPartialConversion(module, target, std::move(patterns))))
signalPassFailure();
diff --git a/mlir/lib/Conversion/OpenACCToLLVM/ACCToLLVMUtils.cpp b/mlir/lib/Conversion/OpenACCToLLVM/ACCToLLVMUtils.cpp
index 99f9a7f622c0a..06bc71bf3c341 100644
--- a/mlir/lib/Conversion/OpenACCToLLVM/ACCToLLVMUtils.cpp
+++ b/mlir/lib/Conversion/OpenACCToLLVM/ACCToLLVMUtils.cpp
@@ -8,10 +8,16 @@
#include "mlir/Conversion/OpenACCToLLVM/ACCToLLVMUtils.h"
-#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/Arith/IR/Arith.h"
#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Interfaces/FunctionInterfaces.h"
#include "llvm/ADT/Hashing.h"
+#include "llvm/ADT/STLExtras.h"
#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/Twine.h"
+
+#include <iterator>
using namespace mlir;
using namespace mlir::acc;
@@ -46,11 +52,10 @@ acc::getFileLineColLoc(Location loc, bool errorOnInvalidLocation) {
StringRef acc::getParentFunctionName(Operation *op) {
if (!op)
return "";
- if (auto parentFuncOp = op->getParentOfType<func::FuncOp>())
- return parentFuncOp.getName();
- if (auto parentFuncOp = op->getParentOfType<LLVM::LLVMFuncOp>())
- return parentFuncOp.getSymName();
- return "";
+ auto funcOp = dyn_cast<FunctionOpInterface>(op);
+ if (!funcOp)
+ funcOp = op->getParentOfType<FunctionOpInterface>();
+ return funcOp ? funcOp.getName() : StringRef();
}
StringRef acc::getParentFunctionName(Value value) {
@@ -67,36 +72,68 @@ StringRef acc::getParentFunctionName(ValueRange values) {
return "";
}
+std::string acc::getInternalGlobalName(StringRef kind, StringRef detail) {
+ SmallString<64> name("acc.");
+ name += kind;
+ if (detail.empty())
+ return name.str().str();
+
+ name += '.';
+ for (unsigned char c : detail)
+ name += (llvm::isAlnum(c) || c == '_' || c == '$' || c == '.') ? c : '_';
+ return name.str().str();
+}
+
/// Creates or reuses a module-internal null-terminated string global and
/// returns the GlobalOp.
static LLVM::GlobalOp getOrCreateGlobalStringOp(Location loc,
OpBuilder &builder,
StringRef name, StringRef value,
- ModuleOp module) {
- if (auto global = module.lookupSymbol<LLVM::GlobalOp>(name))
- return global;
+ Region &globalSymbolRegion,
+ SymbolTable *symbolTable) {
+ SmallString<32> nullTermStr(value);
+ nullTermStr.push_back('\0');
+ StringAttr valueAttr = builder.getStringAttr(nullTermStr);
+
+ // A name says what the global holds, so a global of that name is the one to
+ // reuse as long as it holds this very value. Names come out of detail the
+ // source spells and are not one to one with it, so a name can also lead to a
+ // global holding something else, which a suffix then keeps apart.
+ SmallString<64> uniqueName(name);
+ if (symbolTable) {
+ unsigned suffix = 0;
+ while (Operation *taken = symbolTable->lookup(uniqueName)) {
+ auto global = dyn_cast<LLVM::GlobalOp>(taken);
+ if (global && global.getValueOrNull() == valueAttr)
+ return global;
+ uniqueName.clear();
+ (name + "." + Twine(suffix++)).toVector(uniqueName);
+ }
+ }
// Materialize the global through the incoming builder so that it stays
// tracked when the caller is a dialect conversion rewriter.
OpBuilder::InsertionGuard guard(builder);
- builder.setInsertionPointToStart(module.getBody());
- SmallString<32> nullTermStr(value);
- nullTermStr.push_back('\0');
+ builder.setInsertionPointToStart(&globalSymbolRegion.front());
auto arrayTy = LLVM::LLVMArrayType::get(builder.getI8Type(),
nullTermStr.size_in_bytes());
- return LLVM::GlobalOp::create(builder, loc, arrayTy, /*isConstant=*/true,
- LLVM::Linkage::Internal, name,
- builder.getStringAttr(nullTermStr),
- /*alignment=*/0);
+ auto global =
+ LLVM::GlobalOp::create(builder, loc, arrayTy, /*isConstant=*/true,
+ LLVM::Linkage::Internal, uniqueName, valueAttr,
+ /*alignment=*/0);
+ if (symbolTable)
+ symbolTable->insert(global);
+ return global;
}
Value acc::getOrCreateGlobalString(Location loc, OpBuilder &builder,
StringRef name, StringRef value,
- ModuleOp module) {
+ Region &globalSymbolRegion,
+ SymbolTable *symbolTable) {
Type i64Ty = builder.getI64Type();
Type ptrTy = LLVM::LLVMPointerType::get(builder.getContext());
- LLVM::GlobalOp global =
- getOrCreateGlobalStringOp(loc, builder, name, value, module);
+ LLVM::GlobalOp global = getOrCreateGlobalStringOp(
+ loc, builder, name, value, globalSymbolRegion, symbolTable);
Value globalPtr = LLVM::AddressOfOp::create(builder, loc, global);
Value cst0 = LLVM::ConstantOp::create(builder, loc, i64Ty,
@@ -106,7 +143,9 @@ Value acc::getOrCreateGlobalString(Location loc, OpBuilder &builder,
}
Value acc::createIdent(Location loc, StringRef functionName, OpBuilder &builder,
- ModuleOp module, const ACCRuntimeCallConfig &config) {
+ Region &globalSymbolRegion,
+ const ACCRuntimeCallConfig &config,
+ SymbolTable *symbolTable) {
MLIRContext *ctx = builder.getContext();
Type i32Ty = builder.getI32Type();
Type i64Ty = builder.getI64Type();
@@ -115,7 +154,9 @@ Value acc::createIdent(Location loc, StringRef functionName, OpBuilder &builder,
ctx, {i32Ty, i32Ty, i32Ty, i32Ty, ptrTy});
std::string source;
- std::string sourceGlobalName;
+ // The position the globals describe is what names them, so a global found
+ // under such a name holds this very source string or ident.
+ std::string position;
if (auto fileLineColLoc =
getFileLineColLoc(loc, /*errorOnInvalidLocation=*/false)) {
std::string filename = fileLineColLoc->getFilename().str();
@@ -130,28 +171,29 @@ Value acc::createIdent(Location loc, StringRef functionName, OpBuilder &builder,
source += line + ";";
source += column + ";";
source += ";";
- sourceGlobalName = "loc_";
- sourceGlobalName += line + "_";
- sourceGlobalName += column + "_";
- sourceGlobalName +=
- std::to_string(static_cast<uint64_t>(llvm::hash_value(source)));
+ position = line + "." + column + ".";
+ position += std::to_string(static_cast<uint64_t>(llvm::hash_value(source)));
} else {
source = ";unknown;unknown;0;0;;";
- sourceGlobalName = "loc__";
+ position = "unknown";
}
- std::string identGlobalName = "ident_";
- identGlobalName += sourceGlobalName;
- auto identGlobal = module.lookupSymbol<LLVM::GlobalOp>(identGlobalName);
+ std::string identGlobalName = getInternalGlobalName("ident", position);
+ LLVM::GlobalOp identGlobal =
+ symbolTable ? symbolTable->lookup<LLVM::GlobalOp>(identGlobalName)
+ : LLVM::GlobalOp();
if (!identGlobal) {
LLVM::GlobalOp sourceGlobal = getOrCreateGlobalStringOp(
- loc, builder, sourceGlobalName, source, module);
+ loc, builder, getInternalGlobalName("loc", position), source,
+ globalSymbolRegion, symbolTable);
OpBuilder::InsertionGuard guard(builder);
builder.setInsertionPointAfter(sourceGlobal);
identGlobal = LLVM::GlobalOp::create(
builder, loc, structTy, /*isConstant=*/true, LLVM::Linkage::Internal,
identGlobalName, /*value=*/Attribute(), /*alignment=*/0);
+ if (symbolTable)
+ symbolTable->insert(identGlobal);
Block *block = builder.createBlock(&identGlobal.getInitializerRegion());
builder.setInsertionPointToStart(block);
@@ -169,3 +211,109 @@ Value acc::createIdent(Location loc, StringRef functionName, OpBuilder &builder,
return LLVM::AddressOfOp::create(builder, loc, identGlobal);
}
+
+Value acc::castToI64(Location loc, Value value, OpBuilder &builder) {
+ Type i64Ty = builder.getI64Type();
+ if (value.getType() == i64Ty)
+ return value;
+ if (isa<IndexType>(value.getType()))
+ return arith::IndexCastOp::create(builder, loc, i64Ty, value);
+ unsigned bitwidth = value.getType().getIntOrFloatBitWidth();
+ if (bitwidth > 64)
+ return arith::TruncIOp::create(builder, loc, i64Ty, value);
+ return arith::ExtSIOp::create(builder, loc, i64Ty, value);
+}
+
+Value acc::getAsyncQueue(Location loc, Value asyncOperand, bool asyncOnly,
+ OpBuilder &builder,
+ const ACCRuntimeCallConfig &config) {
+ Type i64Ty = builder.getI64Type();
+ if (asyncOnly)
+ return LLVM::ConstantOp::create(builder, loc, i64Ty,
+ config.getAsyncNoValueRuntimeValue());
+ if (asyncOperand)
+ return castToI64(loc, asyncOperand, builder);
+ return LLVM::ConstantOp::create(builder, loc, i64Ty,
+ config.getAsyncSyncRuntimeValue());
+}
+
+LogicalResult acc::emitWaitCall(Location loc, ValueRange waitOperands,
+ Value asyncQueue, OpBuilder &builder,
+ Region &globalSymbolRegion,
+ SymbolTable &symbolTable,
+ const ACCRuntimeCallConfig &config) {
+ Type i32Ty = builder.getI32Type();
+ Type i64Ty = builder.getI64Type();
+ Type ptrTy = LLVM::LLVMPointerType::get(builder.getContext());
+
+ SmallVector<Value> queues;
+ queues.reserve(waitOperands.size());
+ for (Value waitOperand : waitOperands)
+ queues.push_back(castToI64(loc, waitOperand, builder));
+
+ unsigned size = queues.size();
+ Value waitNum = LLVM::ConstantOp::create(builder, loc, i32Ty, size);
+ Value waitList;
+ if (size == 0) {
+ waitList = LLVM::ZeroOp::create(builder, loc, ptrTy);
+ } else {
+ waitList = LLVM::AllocaOp::create(builder, loc, ptrTy, i64Ty, waitNum);
+ for (auto [index, queue] : llvm::enumerate(queues)) {
+ Value idx = LLVM::ConstantOp::create(builder, loc, i32Ty,
+ static_cast<int64_t>(index));
+ Value elementPtr = LLVM::GEPOp::create(builder, loc, ptrTy, i64Ty,
+ waitList, ArrayRef<Value>{idx});
+ LLVM::StoreOp::create(builder, loc, queue, elementPtr);
+ }
+ }
+
+ StringRef functionName = getParentFunctionName(waitOperands);
+ if (functionName.empty())
+ if (Block *block = builder.getInsertionBlock())
+ functionName = getParentFunctionName(block->getParentOp());
+ Value ident = createIdent(loc, functionName, builder, globalSymbolRegion,
+ config, &symbolTable);
+ Value flags = LLVM::ConstantOp::create(builder, loc, i64Ty, 0);
+ Value deviceType = LLVM::ConstantOp::create(
+ builder, loc, i64Ty, config.getDeviceTypeRuntimeValue(DeviceType::None));
+ Value deviceNum = LLVM::ConstantOp::create(builder, loc, i32Ty, 0);
+
+ return createRuntimeCall(
+ loc, builder, globalSymbolRegion, symbolTable,
+ RuntimeFunction::ACCRTL_tgt_acc_wait, config,
+ {ident, flags, deviceType, deviceNum, waitNum, waitList, asyncQueue});
+}
+
+SmallVector<DeviceType, 3>
+acc::getDeviceTypesByPrecedence(DeviceType deviceType) {
+ SmallVector<DeviceType, 3> deviceTypes;
+ if (deviceType != DeviceType::None && deviceType != DeviceType::Star)
+ deviceTypes.push_back(deviceType);
+ deviceTypes.push_back(DeviceType::Star);
+ deviceTypes.push_back(DeviceType::None);
+ return deviceTypes;
+}
+
+LogicalResult acc::emitGuardedByIfCond(Location loc, Value ifCond,
+ RewriterBase &rewriter,
+ function_ref<LogicalResult()> emitFn) {
+ if (!ifCond)
+ return emitFn();
+
+ Block *parentBlock = rewriter.getInsertionBlock();
+ Block *continueBlock =
+ rewriter.splitBlock(parentBlock, rewriter.getInsertionPoint());
+ Block *thenBlock = rewriter.createBlock(
+ parentBlock->getParent(), std::next(Region::iterator(parentBlock)));
+
+ rewriter.setInsertionPointToEnd(parentBlock);
+ LLVM::CondBrOp::create(rewriter, loc, ifCond, thenBlock, ValueRange{},
+ continueBlock, ValueRange{});
+
+ rewriter.setInsertionPointToStart(thenBlock);
+ LogicalResult result = emitFn();
+ rewriter.setInsertionPointToEnd(thenBlock);
+ LLVM::BrOp::create(rewriter, loc, ValueRange{}, continueBlock);
+ rewriter.setInsertionPointToStart(continueBlock);
+ return result;
+}
diff --git a/mlir/lib/Conversion/OpenACCToLLVM/CMakeLists.txt b/mlir/lib/Conversion/OpenACCToLLVM/CMakeLists.txt
index 1cd092181f7dc..6146300919aa1 100644
--- a/mlir/lib/Conversion/OpenACCToLLVM/CMakeLists.txt
+++ b/mlir/lib/Conversion/OpenACCToLLVM/CMakeLists.txt
@@ -1,5 +1,7 @@
add_mlir_conversion_library(MLIROpenACCToLLVM
ACCAtomicPatterns.cpp
+ ACCDataDirectivePatterns.cpp
+ ACCDataRuntime.cpp
ACCExecutableDirectivePatterns.cpp
ACCToLLVMUtils.cpp
ACCToLLVM.cpp
@@ -15,6 +17,7 @@ add_mlir_conversion_library(MLIROpenACCToLLVM
MLIRComplexDialect
MLIRControlFlowToLLVM
MLIRFuncToLLVM
+ MLIRFunctionInterfaces
MLIRIR
MLIRLLVMCommonConversion
MLIRLLVMDialect
diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCRuntimeUtils.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCRuntimeUtils.cpp
index 405b73a74d95d..49673cfcc8981 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCRuntimeUtils.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCRuntimeUtils.cpp
@@ -11,11 +11,17 @@
#include "mlir/IR/SymbolTable.h"
#include "llvm/Support/ErrorHandling.h"
+#include <cassert>
#include <optional>
using namespace mlir;
using namespace mlir::acc;
+ACCRuntimeCallConfig::ACCRuntimeCallConfig() {
+ populateDialectIdentityDeviceTypeMapping(*this);
+ populateDialectIdentityMapFlagsMapping(*this);
+}
+
StringRef acc::getRuntimeFunctionName(RuntimeFunction fn) {
switch (fn) {
#define ACC_RTL(Enum, Str, ...) \
@@ -43,6 +49,50 @@ LLVM::LLVMFunctionType acc::getRuntimeFunctionType(MLIRContext *ctx,
llvm_unreachable("unknown ACC runtime function");
}
+StringRef acc::getDataDescriptorName(DataDescriptor desc) {
+ switch (desc) {
+#define ACC_DESC_BEGIN(Enum, NameStr, DescKind) \
+ case DataDescriptor::Enum: \
+ return NameStr;
+#include "mlir/Dialect/OpenACC/OpenACCRuntimeDescriptors.def"
+ }
+ llvm_unreachable("unknown ACC data descriptor");
+}
+
+DataDescKind acc::getDataDescriptorKind(DataDescriptor desc) {
+ switch (desc) {
+#define ACC_DESC_BEGIN(Enum, NameStr, DescKind) \
+ case DataDescriptor::Enum: \
+ return DescKind;
+#include "mlir/Dialect/OpenACC/OpenACCRuntimeDescriptors.def"
+ }
+ llvm_unreachable("unknown ACC data descriptor");
+}
+
+LLVM::LLVMStructType acc::getDataDescriptorType(MLIRContext *ctx,
+ DataDescriptor desc,
+ Type baseType) {
+ Type Int8 = IntegerType::get(ctx, 8);
+ Type Int32 = IntegerType::get(ctx, 32);
+ Type Int64 = IntegerType::get(ctx, 64);
+ Type Ptr = LLVM::LLVMPointerType::get(ctx);
+ Type Base = baseType;
+
+ switch (desc) {
+#define ACC_DESC_BEGIN(Enum, NameStr, DescKind) \
+ case DataDescriptor::Enum: { \
+ SmallVector<Type> fields;
+#define ACC_DESC_FIELD(Enum, Field, TypeToken) \
+ assert(TypeToken && "no type for descriptor field " #Field); \
+ fields.push_back(TypeToken);
+#define ACC_DESC_END(Enum) \
+ return LLVM::LLVMStructType::getLiteral(ctx, fields); \
+ }
+#include "mlir/Dialect/OpenACC/OpenACCRuntimeDescriptors.def"
+ }
+ llvm_unreachable("unknown ACC data descriptor");
+}
+
void ACCRuntimeCallConfig::setName(RuntimeFunction fn, StringRef name) {
overrides[fn] = name.str();
}
@@ -78,6 +128,49 @@ int64_t ACCRuntimeCallConfig::getDeviceTypeRuntimeValue(DeviceType type) const {
stringifyDeviceType(type));
}
+void ACCRuntimeCallConfig::setMapFlagRuntimeValue(MapFlags flag,
+ int64_t runtimeValue) {
+ mapFlagRuntimeValues[flag] = runtimeValue;
+}
+
+int64_t ACCRuntimeCallConfig::getMapFlagsRuntimeValue(MapFlags flags) const {
+ int64_t runtimeValue = 0;
+ for (unsigned bit = 0; bit != 32; ++bit) {
+ auto flag = static_cast<MapFlags>(1u << bit);
+ if (!bitEnumContainsAny(flags, flag))
+ continue;
+ auto it = mapFlagRuntimeValues.find(flag);
+ if (it == mapFlagRuntimeValues.end())
+ llvm::report_fatal_error(
+ llvm::Twine("missing OpenACC runtime map-flag mapping for ") +
+ stringifyMapFlags(flag));
+ runtimeValue |= it->second;
+ }
+ return runtimeValue;
+}
+
+void ACCRuntimeCallConfig::setMapFlagsPostProcessFn(MapFlagsPostProcessFn fn) {
+ mapFlagsPostProcessFn = std::move(fn);
+}
+
+MapFlags ACCRuntimeCallConfig::postProcessMapFlags(Operation *mapOp,
+ MapFlags flags) const {
+ if (mapFlagsPostProcessFn)
+ return mapFlagsPostProcessFn(mapOp, flags);
+ return flags;
+}
+
+std::string ACCRuntimeCallConfig::formatMapFlags(MapFlags flags) const {
+ int64_t runtimeValue = getMapFlagsRuntimeValue(flags);
+ std::string rendered = stringifyMapFlags(flags);
+ rendered += " (";
+ rendered += std::to_string(runtimeValue);
+ rendered += " / 0x";
+ rendered += llvm::Twine::utohexstr(runtimeValue).str();
+ rendered += ")";
+ return rendered;
+}
+
void ACCRuntimeCallConfig::setAsyncSyncRuntimeValue(int64_t runtimeValue) {
asyncSyncRuntimeValue = runtimeValue;
}
@@ -101,15 +194,23 @@ void acc::populateDialectIdentityDeviceTypeMapping(
config.setDeviceTypeRuntimeValue(*type, value);
}
+void acc::populateDialectIdentityMapFlagsMapping(ACCRuntimeCallConfig &config) {
+ for (unsigned bit = 0; bit != 32; ++bit) {
+ uint32_t value = 1u << bit;
+ if (std::optional<MapFlags> flag = symbolizeMapFlags(value))
+ config.setMapFlagRuntimeValue(*flag, value);
+ }
+}
+
FailureOr<LLVM::CallOp>
-acc::createRuntimeCall(Location loc, OpBuilder &builder, ModuleOp module,
+acc::createRuntimeCall(Location loc, OpBuilder &builder,
+ Region &globalSymbolRegion, SymbolTable &symbolTable,
RuntimeFunction fn, const ACCRuntimeCallConfig &config,
ArrayRef<Value> arguments) {
MLIRContext *ctx = builder.getContext();
LLVM::LLVMFunctionType fnTy = getRuntimeFunctionType(ctx, fn);
StringRef symbolName = config.getName(fn);
- SymbolTable symbolTable(module);
auto func = symbolTable.lookup<LLVM::LLVMFuncOp>(symbolName);
if (func) {
// An existing declaration with a different signature cannot be called with
@@ -119,8 +220,10 @@ acc::createRuntimeCall(Location loc, OpBuilder &builder, ModuleOp module,
<< "' is already declared with signature "
<< func.getFunctionType() << ", expected " << fnTy;
} else {
- OpBuilder moduleBuilder = OpBuilder::atBlockEnd(module.getBody());
+ OpBuilder moduleBuilder =
+ OpBuilder::atBlockEnd(&globalSymbolRegion.front());
func = LLVM::LLVMFuncOp::create(moduleBuilder, loc, symbolName, fnTy);
+ symbolTable.insert(func);
}
return LLVM::CallOp::create(builder, loc, func, arguments);
diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
index 1bdce5068567f..7f0106b0a5083 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
@@ -537,6 +537,22 @@ static Operation *findCorrespondingDataExit(Value entryResult) {
return exitOps.empty() ? nullptr : exitOps.front();
}
+static std::optional<Location> getMappingExitLoc(Value entryResult) {
+ if (auto mapInfo = entryResult.getDefiningOp<MapInfoOp>())
+ if (std::optional<Location> exitLoc = mapInfo.getExitLoc())
+ return exitLoc;
+ if (Operation *exitOp = findCorrespondingDataExit(entryResult))
+ return exitOp->getLoc();
+ return std::nullopt;
+}
+
+std::optional<Location> getMappingExitLoc(ValueRange dataClauseOperands) {
+ for (Value operand : dataClauseOperands)
+ if (std::optional<Location> exitLoc = getMappingExitLoc(operand))
+ return exitLoc;
+ return std::nullopt;
+}
+
static std::optional<DataClause> getExitDataClause(Operation *exitOp) {
return llvm::TypeSwitch<Operation *, std::optional<DataClause>>(exitOp)
.Case<ACC_DATA_EXIT_OPS>([&](auto exit) { return exit.getDataClause(); })
@@ -718,6 +734,16 @@ MapFlags computeDataClauseMapFlags(Operation *entryOp, bool ptrAndObj) {
return flags;
}
+/// Returns the module \p var lives in.
+static ModuleOp getEnclosingModule(Value var) {
+ if (Operation *def = var.getDefiningOp())
+ return def->getParentOfType<ModuleOp>();
+ if (Region *region = var.getParentRegion())
+ if (Operation *parent = region->getParentOp())
+ return parent->getParentOfType<ModuleOp>();
+ return {};
+}
+
int64_t computeMapInfoSizeBytes(Value var, Type varType, DataDescKind descKind,
ValueRange bounds, const DataLayout &dataLayout,
OpenACCSupport *support) {
@@ -726,12 +752,7 @@ int64_t computeMapInfoSizeBytes(Value var, Type varType, DataDescKind descKind,
if (!bounds.empty() || descKind != DataDescKind::none)
return 0;
- ModuleOp module;
- if (Operation *def = var.getDefiningOp())
- module = def->getParentOfType<ModuleOp>();
- else if (Region *region = var.getParentRegion())
- if (Operation *parent = region->getParentOp())
- module = parent->getParentOfType<ModuleOp>();
+ ModuleOp module = getEnclosingModule(var);
if (!module)
return -1;
@@ -750,6 +771,18 @@ int64_t computeMapInfoSizeBytes(Value var, Type varType, DataDescKind descKind,
return -1;
}
+int64_t computeMapInfoSizeBytes(Value var, Type varType, DataDescKind descKind,
+ ValueRange bounds, OpenACCSupport *support) {
+ ModuleOp module = getEnclosingModule(var);
+ if (!module)
+ return -1;
+ std::optional<DataLayout> dataLayout = getDataLayout(module);
+ if (!dataLayout)
+ return -1;
+ return computeMapInfoSizeBytes(var, varType, descKind, bounds, *dataLayout,
+ support);
+}
+
void populateSourceExtents(ValueRange bounds, ArrayRef<int64_t> shape,
OpBuilder &builder) {
if (shape.size() != bounds.size())
diff --git a/mlir/test/Conversion/OpenACCToLLVM/data-diagnostics.mlir b/mlir/test/Conversion/OpenACCToLLVM/data-diagnostics.mlir
new file mode 100644
index 0000000000000..079f880a5dff6
--- /dev/null
+++ b/mlir/test/Conversion/OpenACCToLLVM/data-diagnostics.mlir
@@ -0,0 +1,13 @@
+// RUN: mlir-opt %s -acc-to-llvm -verify-diagnostics
+
+func.func @wait_devnum(%arg0: !llvm.ptr, %devnum: i64, %queue: i64) {
+ %size = arith.constant 4 : i64
+ %map = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4) descKind(none)
+ mapFlags(to) -> !llvm.ptr
+ // expected-error @below {{not yet implemented: wait clause with a devnum modifier}}
+ // expected-error @below {{failed to legalize operation 'acc.enter_data'}}
+ acc.enter_data dataOperands(%map : !llvm.ptr)
+ wait_devnum(%devnum : i64) wait(%queue : i64)
+ return
+}
diff --git a/mlir/test/Conversion/OpenACCToLLVM/data-runtime.mlir b/mlir/test/Conversion/OpenACCToLLVM/data-runtime.mlir
new file mode 100644
index 0000000000000..19bd764843c02
--- /dev/null
+++ b/mlir/test/Conversion/OpenACCToLLVM/data-runtime.mlir
@@ -0,0 +1,522 @@
+// RUN: mlir-opt %s -acc-to-llvm -split-input-file | FileCheck %s
+
+// Data runtime argument emission preserves every dialect map flag.
+// CHECK-LABEL: llvm.func @all_map_flags
+// CHECK: %[[FLAGS:.*]] = llvm.mlir.constant(8384411 : i64) : i64
+// CHECK: llvm.store %[[FLAGS]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_data_enter
+func.func @all_map_flags(%arg0: !llvm.ptr) {
+ %size = arith.constant 4 : i64
+ %map = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4)
+ descKind(none)
+ mapFlags(to, from, delete, ptr_and_obj, private, literal, implicit,
+ devptr, managed_devptr, no_create, gang_private,
+ worker_private, vector_private, init_zero, device_resident,
+ if_present, present, descriptor, reduction) -> !llvm.ptr
+ acc.enter_data dataOperands(%map : !llvm.ptr)
+ return
+}
+
+// -----
+
+// A normal delete decrements the dynamic reference count and therefore does
+// not force unmapping. Finalize adds DELETE while retaining FROM for copyout.
+// CHECK-LABEL: llvm.func @delete_and_finalize
+// CHECK: %[[SIZE0:.*]] = llvm.mlir.constant(4 : i64) : i64
+// CHECK-NEXT: %[[NONE:.*]] = llvm.mlir.constant(0 : i64) : i64
+// CHECK: llvm.store %[[NONE]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_data_exit
+// CHECK: %[[SIZE1:.*]] = llvm.mlir.constant(4 : i64) : i64
+// CHECK-NEXT: %[[FROM_DELETE:.*]] = llvm.mlir.constant(10 : i64) : i64
+// CHECK: llvm.store %[[FROM_DELETE]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_data_exit
+func.func @delete_and_finalize(%arg0: !llvm.ptr) {
+ %delete = acc.getdeviceptr varPtr(%arg0 : !llvm.ptr) varType(i32)
+ dataClause(acc_delete) structured(false) -> !llvm.ptr
+ acc.exit_data dataOperands(%delete : !llvm.ptr)
+ acc.delete accPtr(%delete : !llvm.ptr) dataClause(acc_delete)
+ structured(false)
+
+ %copyout = acc.getdeviceptr varPtr(%arg0 : !llvm.ptr) varType(i32)
+ dataClause(acc_copyout) structured(false) -> !llvm.ptr
+ acc.exit_data dataOperands(%copyout : !llvm.ptr) finalize
+ acc.copyout accPtr(%copyout : !llvm.ptr) to varPtr(%arg0 : !llvm.ptr)
+ varType(i32) dataClause(acc_copyout) structured(false)
+ return
+}
+
+// -----
+
+// Clause operations derive the same map flags as prepared map_info operations.
+// A zero-initialized create carries INIT_ZERO, and reduction carries TO, FROM,
+// IMPLICIT, and REDUCTION through both calls of a structured data region.
+// CHECK-LABEL: llvm.func @clause_map_flags
+// CHECK-DAG: %[[ZERO:.*]] = llvm.mlir.constant(131072 : i64) : i64
+// CHECK-DAG: llvm.store %[[ZERO]], %{{.*}} : i64, !llvm.ptr
+// CHECK-DAG: %[[REDUCTION:.*]] = llvm.mlir.constant(4194819 : i64) : i64
+// CHECK-DAG: llvm.store %[[REDUCTION]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_data_begin
+// CHECK: %[[REDUCTION_END:.*]] = llvm.mlir.constant(4194819 : i64) : i64
+// CHECK: llvm.store %[[REDUCTION_END]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_data_end
+func.func @clause_map_flags(%arg0: !llvm.ptr, %arg1: !llvm.ptr) {
+ %zero = acc.create varPtr(%arg0 : !llvm.ptr) varType(i32)
+ dataClause(acc_create_zero) -> !llvm.ptr
+ %reduction = acc.copyin varPtr(%arg1 : !llvm.ptr) varType(i32)
+ dataClause(acc_reduction) implicit(true) -> !llvm.ptr
+ acc.data dataOperands(%zero, %reduction : !llvm.ptr, !llvm.ptr) {
+ acc.terminator
+ }
+ acc.delete accPtr(%zero : !llvm.ptr) dataClause(acc_create_zero)
+ acc.copyout accPtr(%reduction : !llvm.ptr) to varPtr(%arg1 : !llvm.ptr)
+ varType(i32) dataClause(acc_reduction) implicit(true)
+ return
+}
+
+// -----
+
+// ifPresent is a property of the update construct and applies to each of its
+// mappings in addition to the direction stated by its clause.
+// CHECK-LABEL: llvm.func @update_if_present
+// CHECK: %[[TO_PRESENT:.*]] = llvm.mlir.constant(524289 : i64) : i64
+// CHECK: llvm.store %[[TO_PRESENT]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_data_update
+// CHECK: %[[FROM_PRESENT:.*]] = llvm.mlir.constant(524290 : i64) : i64
+// CHECK: llvm.store %[[FROM_PRESENT]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_data_update
+func.func @update_if_present(%arg0: !llvm.ptr) {
+ %to = acc.update_device varPtr(%arg0 : !llvm.ptr) varType(i32)
+ structured(false) -> !llvm.ptr
+ acc.update dataOperands(%to : !llvm.ptr) ifPresent
+ %from = acc.getdeviceptr varPtr(%arg0 : !llvm.ptr) varType(i32)
+ dataClause(acc_update_host) structured(false) -> !llvm.ptr
+ acc.update dataOperands(%from : !llvm.ptr) ifPresent
+ acc.update_host accPtr(%from : !llvm.ptr) to varPtr(%arg0 : !llvm.ptr)
+ varType(i32) structured(false)
+ return
+}
+
+// -----
+
+// Bounds are packed in source order. sourceExtent describes the whole source
+// object, and an element stride is converted to bytes.
+// CHECK-LABEL: llvm.func @bounded_map
+// CHECK-DAG: %[[LB:.*]] = llvm.mlir.constant(2 : i64) : i64
+// CHECK-DAG: %[[UB:.*]] = llvm.mlir.constant(6 : i64) : i64
+// CHECK-DAG: %[[SOURCE_EXTENT:.*]] = llvm.mlir.constant(20 : i64) : i64
+// CHECK-DAG: %[[STRIDE:.*]] = llvm.mlir.constant(3 : i64) : i64
+// CHECK-DAG: %[[ELEMENT_SIZE:.*]] = llvm.mlir.constant(4 : i64) : i64
+// CHECK-DAG: %[[BYTE_STRIDE:.*]] = llvm.mul %[[STRIDE]], %[[ELEMENT_SIZE]] : i64
+// CHECK-DAG: llvm.store %[[LB]], %{{.*}} : i64, !llvm.ptr
+// CHECK-DAG: llvm.store %[[UB]], %{{.*}} : i64, !llvm.ptr
+// CHECK-DAG: llvm.store %[[SOURCE_EXTENT]], %{{.*}} : i64, !llvm.ptr
+// CHECK-DAG: llvm.store %[[BYTE_STRIDE]], %{{.*}} : i64, !llvm.ptr
+// CHECK-DAG: %[[RANK:.*]] = llvm.mlir.constant(1 : i8) : i8
+// CHECK-DAG: llvm.insertvalue %[[RANK]], %{{.*}}[1]
+// CHECK-DAG: llvm.call @__tgt_acc_data_begin
+func.func @bounded_map(%arg0: !llvm.ptr) {
+ %c1 = arith.constant 1 : index
+ %c2 = arith.constant 2 : index
+ %c3 = arith.constant 3 : index
+ %c5 = arith.constant 5 : index
+ %c6 = arith.constant 6 : index
+ %c20 = arith.constant 20 : index
+ %bound = acc.bounds lowerbound(%c2 : index) upperbound(%c6 : index)
+ extent(%c5 : index) sourceExtent(%c20 : index)
+ stride(%c3 : index) startIdx(%c1 : index)
+ %map = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(tensor<?xi32>)
+ bounds(%bound) elementSize(4) descKind(openacc)
+ mapFlags(to, from) -> !llvm.ptr
+ acc.data dataOperands(%map : !llvm.ptr) {
+ acc.terminator
+ }
+ return
+}
+
+// -----
+
+// A CFI-backed mapping records the CFI descriptor pointer. With OpenACC
+// bounds, the base descriptor version contains both descriptor-kind bits.
+// CHECK-LABEL: llvm.func @cfi_descriptor
+// CHECK: %[[CFI:.*]] = llvm.mlir.zero : !llvm.struct<(i32, ptr)>
+// CHECK: %[[DESC_PTR:.*]] = llvm.insertvalue %arg1, %[[CFI]][1]
+// cfi | openacc = 1 | 4096.
+// CHECK: %[[VERSION:.*]] = llvm.mlir.constant(4097 : i32) : i32
+// CHECK: %[[VERSIONED:.*]] = llvm.insertvalue %[[VERSION]], %[[DESC_PTR]][0]
+// CHECK: llvm.insertvalue %[[VERSIONED]], %{{.*}}[0]
+// CHECK: llvm.call @__tgt_acc_data_begin
+func.func @cfi_descriptor(%arg0: !llvm.ptr, %arg1: !llvm.ptr) {
+ %c0 = arith.constant 0 : index
+ %c1 = arith.constant 1 : index
+ %c7 = arith.constant 7 : index
+ %c8 = arith.constant 8 : index
+ %bound = acc.bounds lowerbound(%c0 : index) upperbound(%c7 : index)
+ extent(%c8 : index) stride(%c1 : index) startIdx(%c1 : index)
+ %map = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(tensor<?xf64>)
+ desc(%arg1 : !llvm.ptr) bounds(%bound) elementSize(8)
+ descKind(cfi, openacc) mapFlags(to, from, descriptor) -> !llvm.ptr
+ acc.data dataOperands(%map : !llvm.ptr) {
+ acc.terminator
+ }
+ return
+}
+
+// -----
+
+// varPtrPtr is the attachment address and therefore supplies ArgBasePtrs.
+// The descriptor itself is used as the base for PTR_AND_OBJ when there is no
+// attachment address.
+// A CFI descriptor without bounds keeps the CFI-only version, unlike the
+// combined version a mapping with OpenACC bounds states.
+// CHECK-LABEL: llvm.func @mapping_bases
+// CHECK: %[[CFI:.*]] = llvm.mlir.zero : !llvm.struct<(i32, ptr)>
+// CHECK: %[[CFI_PTR:.*]] = llvm.insertvalue %arg2, %[[CFI]][1]
+// CHECK: %[[CFI_VERSION:.*]] = llvm.mlir.constant(1 : i32) : i32
+// CHECK: llvm.insertvalue %[[CFI_VERSION]], %[[CFI_PTR]][0]
+// CHECK: llvm.store %arg1, %{{.*}} : !llvm.ptr, !llvm.ptr
+// CHECK: llvm.store %arg4, %{{.*}} : !llvm.ptr, !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_data_begin
+func.func @mapping_bases(%arg0: !llvm.ptr, %arg1: !llvm.ptr,
+ %arg2: !llvm.ptr, %arg3: !llvm.ptr, %arg4: !llvm.ptr) {
+ %size = arith.constant 8 : i64
+ %attached = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i64)
+ varPtrPtr(%arg1 : !llvm.ptr) desc(%arg2 : !llvm.ptr)
+ size(%size : i64) descKind(cfi)
+ mapFlags(to, ptr_and_obj) -> !llvm.ptr
+ %descriptor = acc.map_info varPtr(%arg3 : !llvm.ptr) varType(i64)
+ desc(%arg4 : !llvm.ptr) size(%size : i64) descKind(cfi)
+ mapFlags(to, ptr_and_obj) -> !llvm.ptr
+ acc.data dataOperands(%attached, %descriptor : !llvm.ptr, !llvm.ptr) {
+ acc.terminator
+ }
+ return
+}
+
+// -----
+
+// A local device-resident object has no host base and is forcibly removed when
+// its structured lifetime ends.
+// CHECK-LABEL: llvm.func @device_resident_exit
+// CHECK: %[[RESIDENT:.*]] = llvm.mlir.constant(262144 : i64) : i64
+// CHECK: llvm.call @__tgt_acc_data_begin
+// CHECK: %[[LOCAL_DELETE:.*]] = llvm.mlir.constant(262152 : i64) : i64
+// CHECK: llvm.store %[[LOCAL_DELETE]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_data_end
+func.func @device_resident_exit(%local: !llvm.ptr) {
+ %size = arith.constant 4 : i64
+ %localMap = acc.map_info varPtr(%local : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4) descKind(none)
+ mapFlags(device_resident) -> !llvm.ptr
+ acc.data dataOperands(%localMap : !llvm.ptr) {
+ acc.terminator
+ }
+ return
+}
+
+// -----
+
+// Runtime variable-name globals disambiguate names that sanitize to the same
+// symbol while reusing a global for repeated names with the same contents.
+// CHECK-DAG: llvm.mlir.global internal constant @acc.var_name.a_b("a-b\00")
+// CHECK-DAG: llvm.mlir.global internal constant @acc.var_name.a_b.0("a_b\00")
+// CHECK-NOT: @acc.var_name.a_b.1
+// CHECK-LABEL: llvm.func @mapping_name_collisions
+// CHECK: llvm.call @__tgt_acc_data_enter
+func.func @mapping_name_collisions(%arg0: !llvm.ptr, %arg1: !llvm.ptr,
+ %arg2: !llvm.ptr) {
+ %size = arith.constant 4 : i64
+ %first = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) name("a-b") descKind(none)
+ mapFlags(to) -> !llvm.ptr
+ %second = acc.map_info varPtr(%arg1 : !llvm.ptr) varType(i32)
+ size(%size : i64) name("a_b") descKind(none)
+ mapFlags(to) -> !llvm.ptr
+ %repeat = acc.map_info varPtr(%arg2 : !llvm.ptr) varType(i32)
+ size(%size : i64) name("a-b") descKind(none)
+ mapFlags(to) -> !llvm.ptr
+ acc.enter_data dataOperands(%first, %second, %repeat
+ : !llvm.ptr, !llvm.ptr, !llvm.ptr)
+ return
+}
+
+// -----
+
+// A structured mapping reports the entry operation at data_begin and its
+// exitLoc at data_end.
+// CHECK-DAG: llvm.mlir.global internal constant @acc.loc.3.2.{{[0-9]+}}(";begin.mlir;mapping_exit_location;3;2;;\00")
+// CHECK-DAG: llvm.mlir.global internal constant @acc.loc.9.4.{{[0-9]+}}(";end.mlir;mapping_exit_location;9;4;;\00")
+// CHECK-LABEL: llvm.func @mapping_exit_location
+// CHECK: llvm.mlir.addressof @acc.ident.3.2.{{[0-9]+}}
+// CHECK: llvm.call @__tgt_acc_data_begin
+// CHECK: llvm.mlir.addressof @acc.ident.9.4.{{[0-9]+}}
+// CHECK: llvm.call @__tgt_acc_data_end
+func.func @mapping_exit_location(%arg0: !llvm.ptr) {
+ %size = arith.constant 4 : i64
+ %map = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) exitLoc(loc("end.mlir":9:4)) descKind(none)
+ mapFlags(to, from) -> !llvm.ptr loc("begin.mlir":3:2)
+ acc.data dataOperands(%map : !llvm.ptr) {
+ acc.terminator
+ }
+ return
+}
+
+// -----
+
+// An empty data region is only a lifetime/control-flow scope. It needs no
+// runtime mapping call, but its body still executes.
+// CHECK-LABEL: llvm.func @empty_data
+// CHECK-NOT: __tgt_acc_data_
+// CHECK: llvm.return
+func.func @empty_data() {
+ acc.data {
+ acc.terminator
+ } defaultAttr(none)
+ return
+}
+
+// -----
+
+// A memref is converted to a multi-field LLVM aggregate. The lowering emits
+// one mapped object per field, so the mapping arrays have that many entries.
+// CHECK-LABEL: llvm.func @memref_fields
+// CHECK-COUNT-5: llvm.extractvalue
+// CHECK: %[[FIVE:.*]] = llvm.mlir.constant(5 : i32) : i32
+// CHECK: %[[ELEMENT_SIZE:.*]] = llvm.mlir.constant(8 : i64) : i64
+// CHECK: %[[MEMREF_DESC:.*]] = llvm.mlir.zero : !llvm.struct<(i32, i8, i64, ptr)>
+// CHECK: %[[RANK:.*]] = llvm.mlir.constant(1 : i8) : i8
+// CHECK: %[[WITH_RANK:.*]] = llvm.insertvalue %[[RANK]], %[[MEMREF_DESC]][1]
+// CHECK: %[[WITH_SIZE:.*]] = llvm.insertvalue %[[ELEMENT_SIZE]], %[[WITH_RANK]][2]
+// CHECK: %[[MEMREF_VERSION:.*]] = llvm.mlir.constant(2 : i32) : i32
+// CHECK: llvm.insertvalue %[[MEMREF_VERSION]], %{{.*}}[0]
+// CHECK: llvm.call @__tgt_acc_data_begin
+func.func @memref_fields(%arg0: memref<?xf64>) {
+ %map = acc.copyin varPtr(%arg0 : memref<?xf64>)
+ dataClause(acc_copyin) -> memref<?xf64>
+ acc.data dataOperands(%map : memref<?xf64>) {
+ acc.terminator
+ }
+ acc.delete accPtr(%map : memref<?xf64>) dataClause(acc_copyin)
+ return
+}
+
+// -----
+
+// Every acc.terminator in a multi-block structured region branches to the
+// continuation after the data-end call.
+// CHECK-LABEL: llvm.func @multiblock_data
+// CHECK: llvm.call @__tgt_acc_data_begin
+// CHECK: llvm.br ^[[BODY:bb[0-9]+]]
+// CHECK: ^[[BODY]]:
+// CHECK: llvm.cond_br %arg1, ^[[LEFT:bb[0-9]+]], ^[[RIGHT:bb[0-9]+]]
+// CHECK: ^[[LEFT]]:
+// CHECK: llvm.br ^[[CONT:bb[0-9]+]]
+// CHECK: ^[[RIGHT]]:
+// CHECK: llvm.br ^[[CONT]]
+// CHECK: ^[[CONT]]:
+// CHECK: llvm.call @__tgt_acc_data_end
+func.func @multiblock_data(%arg0: !llvm.ptr, %cond: i1) {
+ %size = arith.constant 4 : i64
+ %map = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4)
+ descKind(none)
+ mapFlags(to, from) -> !llvm.ptr
+ acc.data dataOperands(%map : !llvm.ptr) {
+ cf.cond_br %cond, ^left, ^right
+ ^left:
+ acc.terminator
+ ^right:
+ acc.terminator
+ }
+ return
+}
+
+// -----
+
+// An object mapped by value is sized from what is passed to the runtime rather
+// than from the stated size of the object it was read from.
+// CHECK-LABEL: llvm.func @literal_size
+// CHECK: %[[SIZE:.*]] = llvm.mlir.constant(8 : i64) : i64
+// CHECK: llvm.store %[[SIZE]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_data_enter
+func.func @literal_size(%arg0: !llvm.ptr) {
+ %size = arith.constant 4 : i64
+ %map = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4) descKind(none)
+ mapFlags(to, literal) -> !llvm.ptr
+ acc.enter_data dataOperands(%map : !llvm.ptr)
+ return
+}
+
+// -----
+
+// A stride already expressed in bytes is copied directly into the descriptor.
+// CHECK-LABEL: llvm.func @byte_stride
+// CHECK: %[[STRIDE:.*]] = llvm.mlir.constant(16 : i64) : i64
+// CHECK-NOT: llvm.mul
+// CHECK: llvm.store %[[STRIDE]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_data_enter
+func.func @byte_stride(%arg0: !llvm.ptr) {
+ %c0 = arith.constant 0 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c16 = arith.constant 16 : index
+ %bound = acc.bounds lowerbound(%c0 : index) upperbound(%c3 : index)
+ extent(%c4 : index) stride(%c16 : index) startIdx(%c0 : index)
+ strideInBytes(true)
+ %map = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(tensor<?xi32>)
+ bounds(%bound) elementSize(4) descKind(openacc)
+ mapFlags(to) -> !llvm.ptr
+ acc.enter_data dataOperands(%map : !llvm.ptr)
+ return
+}
+
+// -----
+
+// Extents retain every full source dimension for a multidimensional section.
+// CHECK-LABEL: llvm.func @multidimensional_extents
+// CHECK-DAG: %[[E5:.*]] = llvm.mlir.constant(5 : i64) : i64
+// CHECK-DAG: %[[E12:.*]] = llvm.mlir.constant(12 : i64) : i64
+// CHECK-DAG: %[[E13:.*]] = llvm.mlir.constant(13 : i64) : i64
+// CHECK: %[[RANK:.*]] = llvm.mlir.constant(4 : i8) : i8
+// CHECK: llvm.store %[[E13]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.store %[[E13]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.store %[[E12]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.store %[[E5]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.insertvalue %[[RANK]], %{{.*}}[1]
+// CHECK: llvm.call @__tgt_acc_data_enter
+func.func @multidimensional_extents(%arg0: !llvm.ptr) {
+ %c0 = arith.constant 0 : index
+ %c1 = arith.constant 1 : index
+ %c4 = arith.constant 4 : index
+ %c5 = arith.constant 5 : index
+ %c11 = arith.constant 11 : index
+ %c12 = arith.constant 12 : index
+ %c13 = arith.constant 13 : index
+ %b0 = acc.bounds lowerbound(%c0 : index) upperbound(%c4 : index)
+ extent(%c5 : index) sourceExtent(%c13 : index)
+ stride(%c1 : index) startIdx(%c0 : index)
+ %b1 = acc.bounds lowerbound(%c0 : index) upperbound(%c4 : index)
+ extent(%c5 : index) sourceExtent(%c13 : index)
+ stride(%c1 : index) startIdx(%c0 : index)
+ %b2 = acc.bounds lowerbound(%c0 : index) upperbound(%c11 : index)
+ extent(%c12 : index) sourceExtent(%c12 : index)
+ stride(%c1 : index) startIdx(%c0 : index)
+ %b3 = acc.bounds lowerbound(%c0 : index) upperbound(%c4 : index)
+ extent(%c5 : index) sourceExtent(%c5 : index)
+ stride(%c1 : index) startIdx(%c0 : index)
+ %map = acc.map_info varPtr(%arg0 : !llvm.ptr)
+ varType(tensor<?x?x?x?xf32>) bounds(%b0, %b1, %b2, %b3)
+ elementSize(4) descKind(openacc) mapFlags(to) -> !llvm.ptr
+ acc.enter_data dataOperands(%map : !llvm.ptr)
+ return
+}
+
+// -----
+
+memref.global @device_global : memref<4xf32> = uninitialized
+
+// A device-resident object reached through a global symbol outlives the region
+// that states it, so its mapping is not turned into a deletion the way the
+// mapping of a local one is.
+// CHECK-LABEL: llvm.func @global_device_resident
+// CHECK: %[[ENTER:.*]] = llvm.mlir.constant(262144 : i64) : i64
+// CHECK: llvm.store %[[ENTER]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_data_begin
+// CHECK: %[[EXIT:.*]] = llvm.mlir.constant(262144 : i64) : i64
+// CHECK: llvm.store %[[EXIT]], %{{.*}} : i64, !llvm.ptr
+// CHECK-NOT: llvm.mlir.constant(262152 : i64) : i64
+// CHECK: llvm.call @__tgt_acc_data_end
+func.func @global_device_resident() {
+ %global = memref.get_global @device_global : memref<4xf32>
+ %map = acc.map_info varPtr(%global : memref<4xf32>) varType(f32)
+ elementSize(4) descKind(none)
+ mapFlags(device_resident) -> memref<4xf32>
+ acc.data dataOperands(%map : memref<4xf32>) {
+ acc.terminator
+ }
+ return
+}
+
+// -----
+
+// Named mappings become string globals whose addresses fill ArgNames; a mapping
+// that states no name leaves a null slot instead, so no second name global is
+// created.
+// CHECK: llvm.mlir.global internal constant @acc.var_name.x("x\00")
+// CHECK-NOT: @acc.var_name.
+// CHECK-LABEL: llvm.func @mapping_names
+// CHECK: %[[NAME:.*]] = llvm.mlir.addressof @acc.var_name.x
+// CHECK: %[[GEP:.*]] = llvm.getelementptr %[[NAME]]
+// CHECK: llvm.store %[[GEP]], %{{.*}} : !llvm.ptr, !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_data_enter
+func.func @mapping_names(%arg0: !llvm.ptr, %arg1: !llvm.ptr) {
+ %size = arith.constant 4 : i64
+ %named = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) name("x") descKind(none)
+ mapFlags(to) -> !llvm.ptr
+ %unnamed = acc.map_info varPtr(%arg1 : !llvm.ptr) varType(i32)
+ size(%size : i64) descKind(none)
+ mapFlags(to) -> !llvm.ptr
+ acc.enter_data dataOperands(%named, %unnamed : !llvm.ptr, !llvm.ptr)
+ return
+}
+
+// -----
+
+// create allocates without a copy, copyin_readonly copies to the device, and
+// attach/detach state the attachment address as PTR_AND_OBJ.
+// CHECK-LABEL: llvm.func @create_readonly_detach
+// CHECK-DAG: %[[TO:.*]] = llvm.mlir.constant(1 : i64) : i64
+// CHECK-DAG: %[[PTR_AND_OBJ:.*]] = llvm.mlir.constant(16 : i64) : i64
+// CHECK-DAG: llvm.store %[[TO]], %{{.*}} : i64, !llvm.ptr
+// CHECK-DAG: llvm.store %[[PTR_AND_OBJ]], %{{.*}} : i64, !llvm.ptr
+// CHECK-NOT: llvm.mlir.constant(3 : i64)
+// CHECK: llvm.call @__tgt_acc_data_begin
+func.func @create_readonly_detach(%arg0: !llvm.ptr, %arg1: !llvm.ptr,
+ %arg2: !llvm.ptr) {
+ %create = acc.create varPtr(%arg0 : !llvm.ptr) varType(i32) -> !llvm.ptr
+ %readonly = acc.copyin varPtr(%arg1 : !llvm.ptr) varType(i32)
+ dataClause(acc_copyin_readonly) -> !llvm.ptr
+ %attach = acc.attach varPtr(%arg2 : !llvm.ptr) varType(i32) -> !llvm.ptr
+ acc.data dataOperands(%create, %readonly, %attach
+ : !llvm.ptr, !llvm.ptr, !llvm.ptr) {
+ acc.terminator
+ }
+ acc.delete accPtr(%create : !llvm.ptr)
+ acc.delete accPtr(%readonly : !llvm.ptr) dataClause(acc_copyin_readonly)
+ acc.detach accPtr(%attach : !llvm.ptr)
+ return
+}
+
+// -----
+
+// Bounds on a memref wrap the memref descriptor in the OpenACC overlay, so the
+// version carries both kinds.
+// CHECK-LABEL: llvm.func @memref_bounded
+// CHECK-COUNT-5: llvm.extractvalue
+// CHECK: %[[ELEMENT_SIZE:.*]] = llvm.mlir.constant(4 : i64) : i64
+// CHECK: %[[MEMREF_DESC:.*]] = llvm.mlir.zero : !llvm.struct<(i32, i8, i64, ptr)>
+// CHECK: %[[RANK:.*]] = llvm.mlir.constant(1 : i8) : i8
+// CHECK: %[[WITH_RANK:.*]] = llvm.insertvalue %[[RANK]], %[[MEMREF_DESC]][1]
+// CHECK: llvm.insertvalue %[[ELEMENT_SIZE]], %[[WITH_RANK]][2]
+// memref | openacc = 2 | 4096.
+// CHECK: %[[VERSION:.*]] = llvm.mlir.constant(4098 : i32) : i32
+// CHECK: llvm.insertvalue %[[VERSION]], %{{.*}}[0]
+// CHECK: llvm.call @__tgt_acc_data_begin
+func.func @memref_bounded(%arg0: memref<?xf32>) {
+ %c0 = arith.constant 0 : index
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %bound = acc.bounds lowerbound(%c0 : index) upperbound(%c3 : index)
+ extent(%c4 : index) stride(%c1 : index) startIdx(%c0 : index)
+ %map = acc.copyin varPtr(%arg0 : memref<?xf32>) varType(tensor<?xf32>)
+ bounds(%bound) dataClause(acc_copyin) -> memref<?xf32>
+ acc.data dataOperands(%map : memref<?xf32>) {
+ acc.terminator
+ }
+ acc.delete accPtr(%map : memref<?xf32>) dataClause(acc_copyin)
+ return
+}
diff --git a/mlir/test/Conversion/OpenACCToLLVM/data.mlir b/mlir/test/Conversion/OpenACCToLLVM/data.mlir
new file mode 100644
index 0000000000000..3a01a01e7810f
--- /dev/null
+++ b/mlir/test/Conversion/OpenACCToLLVM/data.mlir
@@ -0,0 +1,343 @@
+// RUN: mlir-opt %s -acc-to-llvm -split-input-file | FileCheck %s
+
+// A structured data region with a prepared map entry becomes a begin/end pair
+// of runtime mapping calls. The map flags and size are taken from acc.map_info.
+// CHECK-LABEL: llvm.func @data_copy
+// CHECK-NOT: acc.data
+// CHECK-NOT: acc.map_info
+// CHECK-DAG: %[[TOFROM:.*]] = llvm.mlir.constant(3 : i64) : i64
+// CHECK-DAG: %[[SIZE:.*]] = llvm.mlir.constant(40 : i64) : i64
+// CHECK: llvm.call @__tgt_acc_data_begin
+// CHECK: llvm.call @__tgt_acc_data_end
+func.func @data_copy(%arg0: !llvm.ptr) {
+ %size = arith.constant 40 : i64
+ %map = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(f32)
+ size(%size : i64) elementSize(4) name("a")
+ descKind(none) mapFlags(to, from) -> !llvm.ptr
+ acc.data dataOperands(%map : !llvm.ptr) {
+ acc.terminator
+ }
+ return
+}
+
+// -----
+
+// Enter and exit data use the unstructured runtime entry points.
+// CHECK-LABEL: llvm.func @enter_exit
+// CHECK: llvm.call @__tgt_acc_data_enter
+// CHECK: llvm.call @__tgt_acc_data_exit
+func.func @enter_exit(%arg0: !llvm.ptr) {
+ %size = arith.constant 4 : i64
+ %in = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4) name("n")
+ descKind(none) mapFlags(to) -> !llvm.ptr
+ acc.enter_data dataOperands(%in : !llvm.ptr)
+ %out = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4) name("n")
+ descKind(none) mapFlags(from) -> !llvm.ptr
+ acc.exit_data dataOperands(%out : !llvm.ptr)
+ return
+}
+
+// -----
+
+// Update device uses the update runtime entry point.
+// CHECK-LABEL: llvm.func @update_device
+// CHECK: llvm.call @__tgt_acc_data_update
+func.func @update_device(%arg0: !llvm.ptr) {
+ %size = arith.constant 4 : i64
+ %map = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4)
+ descKind(none) mapFlags(to) -> !llvm.ptr
+ acc.update dataOperands(%map : !llvm.ptr)
+ return
+}
+
+// -----
+
+// Data-clause operations can be lowered directly when map_info preparation is
+// not needed. Their clause semantics are packed into the same runtime arrays.
+// CHECK-LABEL: llvm.func @data_clause_ops
+// CHECK-NOT: acc.copyin
+// CHECK-NOT: acc.copyout
+// CHECK-DAG: %[[TOFROM:.*]] = llvm.mlir.constant(3 : i64) : i64
+// CHECK-DAG: %[[SIZE:.*]] = llvm.mlir.constant(8 : i64) : i64
+// CHECK: llvm.call @__tgt_acc_data_begin
+// CHECK: llvm.call @__tgt_acc_data_end
+func.func @data_clause_ops(%arg0: !llvm.ptr) {
+ %copy = acc.copyin varPtr(%arg0 : !llvm.ptr) varType(f64)
+ dataClause(acc_copy) name("a") -> !llvm.ptr
+ acc.data dataOperands(%copy : !llvm.ptr) {
+ acc.terminator
+ }
+ acc.copyout accPtr(%copy : !llvm.ptr) to varPtr(%arg0 : !llvm.ptr)
+ varType(f64) dataClause(acc_copy) name("a")
+ return
+}
+
+// -----
+
+// CHECK-LABEL: llvm.func @update_clause_op
+// CHECK-NOT: acc.update_device
+// CHECK: llvm.call @__tgt_acc_data_update
+func.func @update_clause_op(%arg0: !llvm.ptr) {
+ %device = acc.update_device varPtr(%arg0 : !llvm.ptr) varType(i32)
+ name("n") -> !llvm.ptr
+ acc.update dataOperands(%device : !llvm.ptr)
+ return
+}
+
+// -----
+
+// An async clause names the queue the mapping calls run on, and a wait clause
+// makes them wait for the queues it names first. The wait is not part of the
+// data region, so it is emitted once, before the region is entered.
+// CHECK-LABEL: llvm.func @data_async_wait
+// CHECK: %[[QUEUE:.*]] = llvm.mlir.constant(2 : i64) : i64
+// CHECK: llvm.call @__tgt_acc_wait({{.*}}, %[[QUEUE]])
+// CHECK: llvm.call @__tgt_acc_data_begin({{.*}}, %[[QUEUE]])
+// CHECK: llvm.call @__tgt_acc_data_end({{.*}}, %[[QUEUE]])
+func.func @data_async_wait(%arg0: !llvm.ptr) {
+ %queue = arith.constant 2 : i64
+ %size = arith.constant 40 : i64
+ %map = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(f32)
+ size(%size : i64) elementSize(4) name("a")
+ descKind(none) mapFlags(to, from) -> !llvm.ptr
+ acc.data dataOperands(%map : !llvm.ptr) async(%queue : i64)
+ wait({%queue : i64}) {
+ acc.terminator
+ }
+ return
+}
+
+// -----
+
+// An async clause can be given once per device type. This pass lowers the
+// clauses that apply when no device type is selected, so the queue is the one
+// the default async clause names and the nvidia one is left unused.
+// CHECK-LABEL: llvm.func @data_async_device_type
+// CHECK: %[[QUEUE:.*]] = llvm.mlir.constant(1 : i64) : i64
+// CHECK: llvm.call @__tgt_acc_data_begin({{.*}}, %[[QUEUE]])
+func.func @data_async_device_type(%arg0: !llvm.ptr) {
+ %none = arith.constant 1 : i64
+ %nvidia = arith.constant 7 : i64
+ %size = arith.constant 40 : i64
+ %map = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(f32)
+ size(%size : i64) elementSize(4) name("a")
+ descKind(none) mapFlags(to) -> !llvm.ptr
+ acc.data dataOperands(%map : !llvm.ptr)
+ async(%none : i64, %nvidia : i64 [#acc.device_type<nvidia>]) {
+ acc.terminator
+ }
+ return
+}
+
+// -----
+
+// An if clause guards both mapping calls of a data region: the region body
+// runs either way, the mapping only happens when the condition holds.
+// CHECK-LABEL: llvm.func @data_if
+// CHECK: llvm.cond_br %arg1
+// CHECK: llvm.call @__tgt_acc_data_begin
+// CHECK: llvm.cond_br %arg1
+// CHECK: llvm.call @__tgt_acc_data_end
+func.func @data_if(%arg0: !llvm.ptr, %cond: i1) {
+ %size = arith.constant 40 : i64
+ %map = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(f32)
+ size(%size : i64) elementSize(4) name("a")
+ descKind(none) mapFlags(to, from) -> !llvm.ptr
+ acc.data if(%cond) dataOperands(%map : !llvm.ptr) {
+ acc.terminator
+ }
+ return
+}
+
+// -----
+
+// A wait clause without queues waits for all of them, which the runtime call
+// states with an empty queue list.
+// CHECK-LABEL: llvm.func @enter_data_wait_all
+// CHECK: %[[WAITNUM:.*]] = llvm.mlir.constant(0 : i32) : i32
+// CHECK: %[[WAITLIST:.*]] = llvm.mlir.zero : !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_wait({{.*}}, %[[WAITNUM]], %[[WAITLIST]], {{.*}})
+// CHECK: llvm.call @__tgt_acc_data_enter
+func.func @enter_data_wait_all(%arg0: !llvm.ptr) {
+ %size = arith.constant 4 : i64
+ %map = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4) name("n")
+ descKind(none) mapFlags(to) -> !llvm.ptr
+ acc.enter_data dataOperands(%map : !llvm.ptr) wait
+ return
+}
+
+// -----
+
+// Other entry operations use the same runtime-argument emission and keep the
+// flags of their clause: present, no_create, deviceptr and attach. The address
+// of a device pointer is one the runtime must not look up, so it is handed over
+// in a slot the runtime reads it from.
+// CHECK-LABEL: llvm.func @entry_clause_ops
+// CHECK-NOT: acc.present
+// CHECK-NOT: acc.nocreate
+// CHECK-NOT: acc.deviceptr
+// CHECK-NOT: acc.attach
+// CHECK-DAG: llvm.mlir.constant(1048576 : i64) : i64
+// CHECK-DAG: llvm.mlir.constant(8192 : i64) : i64
+// CHECK-DAG: llvm.mlir.constant(1024 : i64) : i64
+// CHECK-DAG: llvm.mlir.constant(16 : i64) : i64
+// CHECK-DAG: llvm.store %arg2, %{{.*}} : !llvm.ptr, !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_data_begin
+// CHECK: llvm.call @__tgt_acc_data_end
+func.func @entry_clause_ops(%arg0: !llvm.ptr, %arg1: !llvm.ptr,
+ %arg2: !llvm.ptr, %arg3: !llvm.ptr) {
+ %present = acc.present varPtr(%arg0 : !llvm.ptr) varType(f64) -> !llvm.ptr
+ %noCreate = acc.nocreate varPtr(%arg1 : !llvm.ptr) varType(f64)
+ -> !llvm.ptr
+ %devicePtr = acc.deviceptr varPtr(%arg2 : !llvm.ptr) varType(f64)
+ -> !llvm.ptr
+ %attach = acc.attach varPtr(%arg3 : !llvm.ptr) varType(f64) -> !llvm.ptr
+ acc.data dataOperands(%present, %noCreate, %devicePtr, %attach
+ : !llvm.ptr, !llvm.ptr, !llvm.ptr, !llvm.ptr) {
+ acc.terminator
+ }
+ return
+}
+
+// -----
+
+// Each unstructured data construct applies its own if condition to the runtime
+// call, without guarding the surrounding function.
+// CHECK-LABEL: llvm.func @unstructured_data_if
+// CHECK: llvm.cond_br %arg1
+// CHECK: llvm.call @__tgt_acc_data_enter
+// CHECK: llvm.cond_br %arg1
+// CHECK: llvm.call @__tgt_acc_data_update
+// CHECK: llvm.cond_br %arg1
+// CHECK: llvm.call @__tgt_acc_data_exit
+// CHECK: llvm.return
+func.func @unstructured_data_if(%arg0: !llvm.ptr, %cond: i1) {
+ %size = arith.constant 4 : i64
+ %enter = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4) descKind(none)
+ mapFlags(to) -> !llvm.ptr
+ acc.enter_data dataOperands(%enter : !llvm.ptr) if(%cond)
+ %update = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4) descKind(none)
+ mapFlags(from) -> !llvm.ptr
+ acc.update dataOperands(%update : !llvm.ptr) if(%cond)
+ %exit = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4) descKind(none)
+ mapFlags(from) -> !llvm.ptr
+ acc.exit_data dataOperands(%exit : !llvm.ptr) if(%cond)
+ return
+}
+
+// -----
+
+// Data constructs can form one async chain. Every wait precedes the operation
+// it constrains, and each operation receives its own queue.
+// CHECK-LABEL: llvm.func @data_async_chain
+// CHECK: %[[Q100:.*]] = llvm.mlir.constant(100 : i64) : i64
+// CHECK: llvm.call @__tgt_acc_data_begin({{.*}}, %[[Q100]])
+// CHECK: %[[Q200:.*]] = llvm.mlir.constant(200 : i64) : i64
+// CHECK: %[[W100:.*]] = llvm.mlir.constant(100 : i64) : i64
+// CHECK: llvm.store %[[W100]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_wait({{.*}}, %[[Q200]])
+// CHECK: llvm.call @__tgt_acc_data_update({{.*}}, %[[Q200]])
+// CHECK: llvm.call @__tgt_acc_data_end({{.*}}, %[[Q100]])
+// CHECK: %[[Q300:.*]] = llvm.mlir.constant(300 : i64) : i64
+// CHECK: %[[W100_ENTER:.*]] = llvm.mlir.constant(100 : i64) : i64
+// CHECK: llvm.store %[[W100_ENTER]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_wait({{.*}}, %[[Q300]])
+// CHECK: llvm.call @__tgt_acc_data_enter({{.*}}, %[[Q300]])
+// CHECK: %[[Q400:.*]] = llvm.mlir.constant(400 : i64) : i64
+// CHECK: %[[W300:.*]] = llvm.mlir.constant(300 : i64) : i64
+// CHECK: llvm.store %[[W300]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_wait({{.*}}, %[[Q400]])
+// CHECK: llvm.call @__tgt_acc_data_exit({{.*}}, %[[Q400]])
+func.func @data_async_chain(%arg0: !llvm.ptr) {
+ %q100 = arith.constant 100 : i32
+ %q200 = arith.constant 200 : i32
+ %q300 = arith.constant 300 : i32
+ %q400 = arith.constant 400 : i32
+ %size = arith.constant 4 : i64
+ %regionMap = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4) descKind(none)
+ mapFlags(to, from) -> !llvm.ptr
+ acc.data dataOperands(%regionMap : !llvm.ptr) async(%q100 : i32) {
+ %updateMap = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4) descKind(none)
+ mapFlags(from) -> !llvm.ptr
+ acc.update dataOperands(%updateMap : !llvm.ptr) async(%q200 : i32)
+ wait({%q100 : i32})
+ acc.terminator
+ }
+ %enterMap = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4) descKind(none)
+ mapFlags(to) -> !llvm.ptr
+ acc.enter_data dataOperands(%enterMap : !llvm.ptr) async(%q300 : i32)
+ wait(%q100 : i32)
+ %exitMap = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4) descKind(none)
+ mapFlags(from) -> !llvm.ptr
+ acc.exit_data dataOperands(%exitMap : !llvm.ptr) async(%q400 : i32)
+ wait(%q300 : i32)
+ return
+}
+
+// -----
+
+// Async queue operands are normalized to the runtime's i64 queue type.
+// CHECK-LABEL: llvm.func @data_async_integer_widths
+// CHECK: %[[Q32:.*]] = llvm.sext %arg1 : i32 to i64
+// CHECK: llvm.call @__tgt_acc_data_enter({{.*}}, %[[Q32]])
+// CHECK: %[[Q16:.*]] = llvm.sext %arg2 : i16 to i64
+// CHECK: llvm.call @__tgt_acc_data_exit({{.*}}, %[[Q16]])
+func.func @data_async_integer_widths(%arg0: !llvm.ptr, %q32: i32, %q16: i16) {
+ %size = arith.constant 4 : i64
+ %enter = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4) descKind(none)
+ mapFlags(to) -> !llvm.ptr
+ acc.enter_data dataOperands(%enter : !llvm.ptr) async(%q32 : i32)
+ %exit = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4) descKind(none)
+ mapFlags(from) -> !llvm.ptr
+ acc.exit_data dataOperands(%exit : !llvm.ptr) async(%q16 : i16)
+ return
+}
+
+// -----
+
+// An async clause without a queue uses the runtime's no-value sentinel.
+// CHECK-LABEL: llvm.func @data_async_noval
+// CHECK: %[[NOVAL:.*]] = llvm.mlir.constant(-4 : i64) : i64
+// CHECK: llvm.call @__tgt_acc_data_begin({{.*}}, %[[NOVAL]])
+// CHECK: llvm.call @__tgt_acc_data_end({{.*}}, %[[NOVAL]])
+func.func @data_async_noval(%arg0: !llvm.ptr) {
+ %size = arith.constant 4 : i64
+ %map = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4) descKind(none)
+ mapFlags(to, from) -> !llvm.ptr
+ acc.data dataOperands(%map : !llvm.ptr) async {
+ acc.terminator
+ }
+ return
+}
+
+// -----
+
+// Update waits for the named queues before the mapping call.
+// CHECK-LABEL: llvm.func @update_wait
+// CHECK: %[[QUEUE:.*]] = llvm.mlir.constant(7 : i64) : i64
+// CHECK: llvm.store %[[QUEUE]], %{{.*}} : i64, !llvm.ptr
+// CHECK: llvm.call @__tgt_acc_wait
+// CHECK: llvm.call @__tgt_acc_data_update
+func.func @update_wait(%arg0: !llvm.ptr) {
+ %q = arith.constant 7 : i32
+ %size = arith.constant 4 : i64
+ %map = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4) descKind(none)
+ mapFlags(from) -> !llvm.ptr
+ acc.update dataOperands(%map : !llvm.ptr) wait({%q : i32})
+ return
+}
diff --git a/mlir/test/Conversion/OpenACCToLLVM/init-shutdown-set.mlir b/mlir/test/Conversion/OpenACCToLLVM/init-shutdown-set.mlir
index babb4abf1d5e2..4c37ea91d67d1 100644
--- a/mlir/test/Conversion/OpenACCToLLVM/init-shutdown-set.mlir
+++ b/mlir/test/Conversion/OpenACCToLLVM/init-shutdown-set.mlir
@@ -7,7 +7,7 @@
// the device number widened to i64. With the default ACCRuntimeCallConfig
// mapping (dialect ordinal), nvidia is 5.
// CHECK: %[[NVIDIA:.*]] = llvm.mlir.constant(5 : i64) : i64
-// CHECK: %[[IDENT:.*]] = llvm.mlir.addressof @[[ID:ident_[^ ]+]]
+// CHECK: %[[IDENT:.*]] = llvm.mlir.addressof @[[ID:acc\.ident\.[^ ]+]]
// CHECK: %[[FLAGS:.*]] = llvm.mlir.constant(0 : i64) : i64
// CHECK: %[[DEVNUM:.*]] = llvm.mlir.constant(2 : i64) : i64
// CHECK: llvm.call @__tgt_acc_init(%[[IDENT]], %[[FLAGS]], %[[NVIDIA]], %[[DEVNUM]]) : (!llvm.ptr, i64, i64, i64) -> ()
@@ -93,15 +93,15 @@ module {
// The ident is a constant global whose source field points at the location
// string. Call sites take the address of that ident, not of the string.
-// CHECK: llvm.mlir.global internal constant @[[$SRC:loc_10_1_[0-9]+]](";init-shutdown-set.mlir;test_init_with_loc;10;1;;\00")
-// CHECK: llvm.mlir.global internal constant @[[$IDENT:ident_loc_10_1_[0-9]+]]() {{.*}} : !llvm.struct<(i32, i32, i32, i32, ptr)> {
+// CHECK: llvm.mlir.global internal constant @acc.loc.[[$POS:10\.1\.[0-9]+]](";init-shutdown-set.mlir;test_init_with_loc;10;1;;\00")
+// CHECK: llvm.mlir.global internal constant @acc.ident.[[$POS]]() {{.*}} : !llvm.struct<(i32, i32, i32, i32, ptr)> {
// CHECK: llvm.mlir.zero : !llvm.struct<(i32, i32, i32, i32, ptr)>
-// CHECK: llvm.mlir.addressof @[[$SRC]]
+// CHECK: llvm.mlir.addressof @acc.loc.[[$POS]]
// CHECK: llvm.getelementptr
// CHECK: llvm.insertvalue {{.*}}[4]
// CHECK: llvm.return
// CHECK-LABEL: llvm.func @test_init_with_loc
-// CHECK: llvm.mlir.addressof @[[$IDENT]]
+// CHECK: llvm.mlir.addressof @acc.ident.[[$POS]]
// CHECK: llvm.call @__tgt_acc_init
#loc = loc("init-shutdown-set.mlir":10:1)
@@ -116,15 +116,15 @@ module {
// Operations without file:line information fall back to an unknown ident.
-// CHECK: llvm.mlir.global internal constant @loc__(";unknown;unknown;0;0;;\00")
-// CHECK: llvm.mlir.global internal constant @ident_loc__() {{.*}} : !llvm.struct<(i32, i32, i32, i32, ptr)> {
+// CHECK: llvm.mlir.global internal constant @acc.loc.unknown(";unknown;unknown;0;0;;\00")
+// CHECK: llvm.mlir.global internal constant @acc.ident.unknown() {{.*}} : !llvm.struct<(i32, i32, i32, i32, ptr)> {
// CHECK: llvm.mlir.zero : !llvm.struct<(i32, i32, i32, i32, ptr)>
-// CHECK: llvm.mlir.addressof @loc__
+// CHECK: llvm.mlir.addressof @acc.loc.unknown
// CHECK: llvm.getelementptr
// CHECK: llvm.insertvalue {{.*}}[4]
// CHECK: llvm.return
// CHECK-LABEL: llvm.func @test_init_unknown_loc
-// CHECK: llvm.mlir.addressof @ident_loc__
+// CHECK: llvm.mlir.addressof @acc.ident.unknown
// CHECK: llvm.call @__tgt_acc_init
module {
@@ -139,12 +139,12 @@ module {
// Two operations at the same line and column of different files must not share
// a location global, otherwise the ident would name the wrong file.
-// CHECK-DAG: llvm.mlir.global internal constant @[[$LOC_A:loc_7_3_[0-9]+]](";a.mlir;test_distinct_files;7;3;;\00")
-// CHECK-DAG: llvm.mlir.global internal constant @[[$LOC_B:loc_7_3_[0-9]+]](";b.mlir;test_distinct_files;7;3;;\00")
+// CHECK-DAG: llvm.mlir.global internal constant @acc.loc.[[$POS_A:7\.3\.[0-9]+]](";a.mlir;test_distinct_files;7;3;;\00")
+// CHECK-DAG: llvm.mlir.global internal constant @acc.loc.[[$POS_B:7\.3\.[0-9]+]](";b.mlir;test_distinct_files;7;3;;\00")
// CHECK-LABEL: llvm.func @test_distinct_files
-// CHECK: llvm.mlir.addressof @ident_[[$LOC_A]]
+// CHECK: llvm.mlir.addressof @acc.ident.[[$POS_A]]
// CHECK: llvm.call @__tgt_acc_init
-// CHECK: llvm.mlir.addressof @ident_[[$LOC_B]]
+// CHECK: llvm.mlir.addressof @acc.ident.[[$POS_B]]
// CHECK: llvm.call @__tgt_acc_shutdown
module {
diff --git a/mlir/test/Conversion/OpenACCToLLVM/runtime-declaration-mismatch.mlir b/mlir/test/Conversion/OpenACCToLLVM/runtime-declaration-mismatch.mlir
index 453bc56871b10..43d08c5b9fae0 100644
--- a/mlir/test/Conversion/OpenACCToLLVM/runtime-declaration-mismatch.mlir
+++ b/mlir/test/Conversion/OpenACCToLLVM/runtime-declaration-mismatch.mlir
@@ -24,3 +24,24 @@ module {
return
}
}
+
+// -----
+
+// Data mapping calls use the same checked declaration path.
+module {
+ llvm.func @__tgt_acc_data_begin(!llvm.ptr, i64)
+ func.func @mismatched_data(%arg0: !llvm.ptr) {
+ %size = arith.constant 4 : i64
+ // The mapping call is reported at the operand that states the mapping,
+ // which is where the runtime arguments are emitted.
+ // expected-error @below {{OpenACC runtime function '__tgt_acc_data_begin' is already declared with signature}}
+ %map = acc.map_info varPtr(%arg0 : !llvm.ptr) varType(i32)
+ size(%size : i64) elementSize(4) descKind(none)
+ mapFlags(to) -> !llvm.ptr
+ // expected-error @below {{failed to legalize operation 'acc.data'}}
+ acc.data dataOperands(%map : !llvm.ptr) {
+ acc.terminator
+ }
+ return
+ }
+}
diff --git a/mlir/test/Conversion/OpenACCToLLVM/wait.mlir b/mlir/test/Conversion/OpenACCToLLVM/wait.mlir
index 87aae3fb27bb7..3533714f0a0a5 100644
--- a/mlir/test/Conversion/OpenACCToLLVM/wait.mlir
+++ b/mlir/test/Conversion/OpenACCToLLVM/wait.mlir
@@ -102,15 +102,15 @@ module {
// The ident is a constant global whose source field points at the location
// string. Call sites take the address of that ident, not of the string.
-// CHECK: llvm.mlir.global internal constant @[[$SRC:loc_5_3_[0-9]+]](";wait.mlir;test_wait_with_loc;5;3;;\00")
-// CHECK: llvm.mlir.global internal constant @[[$IDENT:ident_loc_5_3_[0-9]+]]() {{.*}} : !llvm.struct<(i32, i32, i32, i32, ptr)> {
+// CHECK: llvm.mlir.global internal constant @acc.loc.[[$POS:5\.3\.[0-9]+]](";wait.mlir;test_wait_with_loc;5;3;;\00")
+// CHECK: llvm.mlir.global internal constant @acc.ident.[[$POS]]() {{.*}} : !llvm.struct<(i32, i32, i32, i32, ptr)> {
// CHECK: llvm.mlir.zero : !llvm.struct<(i32, i32, i32, i32, ptr)>
-// CHECK: llvm.mlir.addressof @[[$SRC]]
+// CHECK: llvm.mlir.addressof @acc.loc.[[$POS]]
// CHECK: llvm.getelementptr
// CHECK: llvm.insertvalue {{.*}}[4]
// CHECK: llvm.return
// CHECK-LABEL: llvm.func @test_wait_with_loc
-// CHECK: llvm.mlir.addressof @[[$IDENT]]
+// CHECK: llvm.mlir.addressof @acc.ident.[[$POS]]
// CHECK: llvm.call @__tgt_acc_wait
#loc = loc("wait.mlir":5:3)
@@ -125,15 +125,15 @@ module {
// Operations without file:line information fall back to an unknown ident.
-// CHECK: llvm.mlir.global internal constant @loc__(";unknown;unknown;0;0;;\00")
-// CHECK: llvm.mlir.global internal constant @ident_loc__() {{.*}} : !llvm.struct<(i32, i32, i32, i32, ptr)> {
+// CHECK: llvm.mlir.global internal constant @acc.loc.unknown(";unknown;unknown;0;0;;\00")
+// CHECK: llvm.mlir.global internal constant @acc.ident.unknown() {{.*}} : !llvm.struct<(i32, i32, i32, i32, ptr)> {
// CHECK: llvm.mlir.zero : !llvm.struct<(i32, i32, i32, i32, ptr)>
-// CHECK: llvm.mlir.addressof @loc__
+// CHECK: llvm.mlir.addressof @acc.loc.unknown
// CHECK: llvm.getelementptr
// CHECK: llvm.insertvalue {{.*}}[4]
// CHECK: llvm.return
// CHECK-LABEL: llvm.func @test_wait_unknown_loc
-// CHECK: llvm.mlir.addressof @ident_loc__
+// CHECK: llvm.mlir.addressof @acc.ident.unknown
// CHECK: llvm.call @__tgt_acc_wait
module {
diff --git a/mlir/unittests/Dialect/OpenACC/CMakeLists.txt b/mlir/unittests/Dialect/OpenACC/CMakeLists.txt
index 676b87161c114..0fe641c2da556 100644
--- a/mlir/unittests/Dialect/OpenACC/CMakeLists.txt
+++ b/mlir/unittests/Dialect/OpenACC/CMakeLists.txt
@@ -2,6 +2,7 @@ add_mlir_unittest(MLIROpenACCTests
OpenACCCGOpsTest.cpp
OpenACCOpsTest.cpp
OpenACCOpsInterfacesTest.cpp
+ OpenACCRuntimeUtilsTest.cpp
OpenACCTypeInterfacesTest.cpp
OpenACCUtilsCGTest.cpp
OpenACCUtilsGPUTest.cpp
diff --git a/mlir/unittests/Dialect/OpenACC/OpenACCRuntimeUtilsTest.cpp b/mlir/unittests/Dialect/OpenACC/OpenACCRuntimeUtilsTest.cpp
new file mode 100644
index 0000000000000..b85217bbe24e6
--- /dev/null
+++ b/mlir/unittests/Dialect/OpenACC/OpenACCRuntimeUtilsTest.cpp
@@ -0,0 +1,119 @@
+//===- OpenACCRuntimeUtilsTest.cpp - OpenACC runtime utility tests --------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/OpenACCRuntimeUtils.h"
+#include "mlir/IR/MLIRContext.h"
+#include "gtest/gtest.h"
+
+#include <optional>
+
+using namespace mlir;
+using namespace mlir::acc;
+
+TEST(OpenACCRuntimeCallConfigTest, UsesDialectDeviceTypeEncodingsByDefault) {
+ ACCRuntimeCallConfig config;
+ for (uint32_t value = 0; value <= getMaxEnumValForDeviceType(); ++value) {
+ std::optional<DeviceType> deviceType = symbolizeDeviceType(value);
+ if (deviceType)
+ EXPECT_EQ(config.getDeviceTypeRuntimeValue(*deviceType), value);
+ }
+}
+
+TEST(OpenACCRuntimeCallConfigTest, UsesDialectMapFlagEncodingsByDefault) {
+ ACCRuntimeCallConfig config;
+ for (unsigned bit = 0; bit != 32; ++bit) {
+ uint32_t value = 1u << bit;
+ std::optional<MapFlags> flag = symbolizeMapFlags(value);
+ if (flag)
+ EXPECT_EQ(config.getMapFlagsRuntimeValue(*flag), value);
+ }
+}
+
+TEST(OpenACCRuntimeCallConfigTest, CombinesConfiguredMapFlagEncodings) {
+ ACCRuntimeCallConfig config;
+ config.setMapFlagRuntimeValue(MapFlags::to, 0x40);
+ config.setMapFlagRuntimeValue(MapFlags::from, 0x100);
+
+ EXPECT_EQ(config.getMapFlagsRuntimeValue(MapFlags::to | MapFlags::from),
+ 0x140);
+}
+
+TEST(OpenACCRuntimeCallConfigTest, PostProcessesMapFlags) {
+ ACCRuntimeCallConfig config;
+ EXPECT_EQ(config.postProcessMapFlags(nullptr, MapFlags::to), MapFlags::to);
+
+ Operation *seenMapOp = nullptr;
+ config.setMapFlagsPostProcessFn([&](Operation *mapOp, MapFlags flags) {
+ seenMapOp = mapOp;
+ return flags | MapFlags::delete_;
+ });
+
+ EXPECT_EQ(config.postProcessMapFlags(nullptr, MapFlags::from),
+ MapFlags::from | MapFlags::delete_);
+ EXPECT_EQ(seenMapOp, nullptr);
+}
+
+TEST(OpenACCRuntimeCallConfigTest, FormatsConfiguredMapFlags) {
+ ACCRuntimeCallConfig config;
+ config.setMapFlagRuntimeValue(MapFlags::to, 0x40);
+ config.setMapFlagRuntimeValue(MapFlags::from, 0x100);
+
+ EXPECT_EQ(config.formatMapFlags(MapFlags::to | MapFlags::from),
+ "to,from (320 / 0x140)");
+}
+
+TEST(OpenACCDataDescriptorTest, StatesTheKindOfEveryDescriptor) {
+ EXPECT_EQ(getDataDescriptorKind(DataDescriptor::AccDataDescGeneric),
+ DataDescKind::none);
+ EXPECT_EQ(getDataDescriptorKind(DataDescriptor::AccDataDescCFI),
+ DataDescKind::cfi);
+ EXPECT_EQ(getDataDescriptorKind(DataDescriptor::AccDataDescMemRef),
+ DataDescKind::memref);
+ EXPECT_EQ(getDataDescriptorKind(DataDescriptor::AccDataDescOpenACC),
+ DataDescKind::openacc);
+ EXPECT_EQ(getDataDescriptorName(DataDescriptor::AccDataDescMemRef),
+ "AccDataDescMemRef");
+}
+
+TEST(OpenACCDataDescriptorTest, BuildsTheDeclaredLayouts) {
+ MLIRContext ctx;
+ ctx.loadDialect<LLVM::LLVMDialect>();
+ Type i8Ty = IntegerType::get(&ctx, 8);
+ Type i32Ty = IntegerType::get(&ctx, 32);
+ Type i64Ty = IntegerType::get(&ctx, 64);
+ Type ptrTy = LLVM::LLVMPointerType::get(&ctx);
+
+ EXPECT_EQ(
+ getDataDescriptorType(&ctx, DataDescriptor::AccDataDescGeneric).getBody(),
+ ArrayRef<Type>({i32Ty, i32Ty}));
+ EXPECT_EQ(
+ getDataDescriptorType(&ctx, DataDescriptor::AccDataDescCFI).getBody(),
+ ArrayRef<Type>({i32Ty, ptrTy}));
+ EXPECT_EQ(
+ getDataDescriptorType(&ctx, DataDescriptor::AccDataDescMemRef).getBody(),
+ ArrayRef<Type>({i32Ty, i8Ty, i64Ty, ptrTy}));
+
+ Type baseTy = getDataDescriptorType(&ctx, DataDescriptor::AccDataDescMemRef);
+ EXPECT_EQ(
+ getDataDescriptorType(&ctx, DataDescriptor::AccDataDescOpenACC, baseTy)
+ .getBody(),
+ ArrayRef<Type>({baseTy, i8Ty, i64Ty, ptrTy, ptrTy, ptrTy, ptrTy, ptrTy}));
+}
+
+TEST(OpenACCDataDescriptorTest, NumbersFieldsInDeclarationOrder) {
+ EXPECT_EQ(getDataDescriptorFieldIndex(AccDataDescMemRefField::Version), 0);
+ EXPECT_EQ(getDataDescriptorFieldIndex(AccDataDescMemRefField::Rank), 1);
+ EXPECT_EQ(getDataDescriptorFieldIndex(AccDataDescMemRefField::ElementSize),
+ 2);
+ EXPECT_EQ(
+ getDataDescriptorFieldIndex(AccDataDescMemRefField::MemRefDescriptor), 3);
+
+ EXPECT_EQ(getDataDescriptorFieldIndex(AccDataDescOpenACCField::Base), 0);
+ EXPECT_EQ(getDataDescriptorFieldIndex(AccDataDescOpenACCField::StartIndices),
+ 7);
+}
More information about the Mlir-commits
mailing list