[flang-commits] [flang] [Flang][MLIR][OpenMP] Create a deferred declare target marking process for Bridge.cpp (PR #78502)

[Sergio Afonso via flang-commits]

================
@@ -0,0 +1,124 @@
+<!--===- docs/OpenMP-declare-target.md
+
+   Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+   See https://llvm.org/LICENSE.txt for license information.
+   SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+-->
+
+# Introduction to Declare Target
+
+In OpenMP `declare target` is a directive that can be applied to a function or variable (primarily global) to notate to the compiler that it should be generated in a particular devices environment. In essence whether something should be emitted for host or device, or both. An example of its usage for both data and functions can be seen below.
+
+```Fortran
+module test_0
+    integer :: sp = 0
+!$omp declare target link(sp)
+end module test_0
+
+program main
+    use test_0
+!$omp target map(tofrom:sp)
+    sp = 1
+!$omp end target
+end program
+```
+
+In the above example, we created a variable in a seperate module, mark it as `declare target` and then map it, embedding it into the device IR and assigning to it. 
+
+
+```Fortran
+function func_t_device() result(i)
+    !$omp declare target to(func_t_device) device_type(nohost)
+        INTEGER :: I
+        I = 1
+end function func_t_device
+
+program main
+!$omp target
+    call func_t_device()
+!$omp end target
+end program
+```
+
+In the above example, we are stating that a function is required on device utilising `declare target`, and that we will not be utilising it on host, so we are in theory free to remove or ignore it. A user could also in this case, leave off the `declare target` from the function and it would be implicitly marked `declare target any` (for both host and device), as it's been utilised within a target region.
+
+# Declare Target as represented in the OpenMP Dialect
+
+In the OpenMP Dialect `declare target` is not represented by a specific `operation` instead it's a OpenMP dialect specific `attribute` that can be applied to any operation in any dialect. This helps to simplify the utilisation of it, instead of replacing or modifying existing global or function `operations` in a dialect it applies to it as extra metadata that the lowering can use in different ways as is neccesary. 
+
+The `attribute` is composed of multiple fields representing the clauses you would find on the `declare target` directive i.e. device type (`nohost`, `any`, `host`) or the capture clause (`link` or `to`). A small example of `declare target` applied to an Fortran `real` can be found below:
+
+```
+fir.global internal @_QFEi {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to)>} : f32 {
+    %0 = fir.undefined f32
+    fir.has_value %0 : f32
+}
+```
+
+This would look similar for function style `operations`.
+
+The application and access of this attribute is aided by an OpenMP Dialect MLIR Interface named `DeclareTargetInterface`, which can be utilised on operations to access the appropriate interface functions, e.g.:
+
+```C++
+auto declareTargetGlobal = llvm::dyn_cast<mlir::omp::DeclareTargetInterface>(Op.getOperation());
+declareTargetGlobal.isDeclareTarget();
+```
+
+# Declare Target Fortran OpenMP Lowering
+
+The initial lowering of `declare target` to MLIR for both use-cases is done inside of the usual OpenMP lowering in flang/lib/Lower/OpenMP.cpp, however some direct calls to `declare target` related functions from Flang's Bridge in flang/lib/Lower/Bridge.cpp are made.
+
+The marking of operations with the declare target attribute happens in two phases, the second one optional contingent on the first failing to apply the attribute due to the operation not being generated yet, the main notable case this occurs currently is when a Fortran function interface has been marked. 
+
+The initial phase happens when the declare target directive and it's clauses are initially processed, with the primary data gathering for the directive and clause happening in a function called `getDeclareTargetInfo` which is then used to feed the `markDeclareTarget` function which does the actual marking utilising the `DeclareTargetInterface`, if it encounters something that has been marked twice over multiple directives with two differing device types (e.g. `host`, `nohost`), then it will swap the device type to `any`.
+
+Whenever we invoke `genFIR` on an `OpenMPDeclarativeConstruct` from Bridge, we are also invoking another function 
+called `gatherOpenMPDeferredDeclareTargets` which gathers information relevant to the application of the `declare target` attribute (symbol that it should be applied to, device type clause, and capture clause) when processing `declare target` declarations, storing the data in a vector that is part of Bridge's instantiation of the `AbstractConverter`. This data is only stored if we encounter a function or variable symbol that does not have an operation instantiated for it yet, unfortunately this cannot happen as part of the initial marking as we must store this data in Bridge and only have access to the abstract version of the converter via the OpenMP lowering. 
+
+This information is used in the second phase, which is a form of deferred processing of the `declare target` marked operations that have delayed generation, this is done via the function `markOpenMPDeferredDeclareTargetFunctions` which is called from Bridge at the end of the lowering process allowing us to mark those where possible. It iterates over the data gathered by `gatherOpenMPDeferredDeclareTargets` checking if any of the recorded symbols have now had their corresponding operations instantiated and applying where possible utilising `markDeclareTarget`.
----------------
skatrak wrote:

```suggestion
The information produced by the first phase is used in the second phase, which is a form of deferred processing of the `declare target` marked operations that have delayed generation. This is done via the function `markOpenMPDeferredDeclareTargetFunctions`, which is called from the lowering bridge at the end of the process. It iterates over the data previously gathered checking if any of the recorded symbols have now had their corresponding operations instantiated and applying the declare target attribute where possible utilising `markDeclareTarget`.
```

https://github.com/llvm/llvm-project/pull/78502