[flang-commits] [flang] [Flang][MLIR][OpenMP] Create a deferred declare target marking process for Bridge.cpp (PR #78502)

[Sergio Afonso via flang-commits]

================
@@ -0,0 +1,259 @@
+<!--===- docs/OpenMP-declare-target.md
+
+   Part of the LLVM Project, under the Apache License v2.0 with LLVM
+   Exceptions.
+   See https://llvm.org/LICENSE.txt for license information.
+   SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+-->
+
+# Introduction to Declare Target
+
+In OpenMP `declare target` is a directive that can be applied to a function or
+variable (primarily global) to notate to the compiler that it should be 
+generated in a particular device's environment. In essence whether something 
+should be emitted for host or device, or both. An example of its usage for 
+both data and functions can be seen below.
+
+```Fortran
+module test_0
+    integer :: sp = 0
+!$omp declare target link(sp)
+end module test_0
+
+program main
+    use test_0
+!$omp target map(tofrom:sp)
+    sp = 1
+!$omp end target
+end program
+```
+
+In the above example, we create a variable in a separate module, mark it 
+as `declare target` and then map it, embedding it into the device IR and 
+assigning to it. 
+
+
+```Fortran
+function func_t_device() result(i)
+    !$omp declare target to(func_t_device) device_type(nohost)
+        INTEGER :: I
+        I = 1
+end function func_t_device
+
+program main
+!$omp target
+    call func_t_device()
+!$omp end target
+end program
+```
+
+In the above example, we are stating that a function is required on device
+utilising `declare target`, and that we will not be utilising it on host, 
+so we are in theory free to remove or ignore it there. A user could also 
+in this case, leave off the `declare target` from the function and it 
+would be implicitly marked `declare target any` (for both host and device), 
+as it's been utilised within a target region.
+
+# Declare Target as represented in the OpenMP Dialect
+
+In the OpenMP Dialect `declare target` is not represented by a specific 
+`operation`. Instead it's a OpenMP dialect specific `attribute` that can be 
+applied to any operation in any dialect. This helps to simplify the 
+utilisation of it, instead of replacing or modifying existing global or 
+function `operations` in a dialect it applies to it as extra metadata that 
+the lowering can use in different ways as is necesary. 
+
+The `attribute` is composed of multiple fields representing the clauses you 
+would find on the `declare target` directive i.e. device type (`nohost`, 
+`any`, `host`) or the capture clause (`link` or `to`). A small example of 
+`declare target` applied to a Fortran `real` can be found below:
+
+```
+fir.global internal @_QFEi {omp.declare_target = 
+#omp.declaretarget<device_type = (any), capture_clause = (to)>} : f32 {
+    %0 = fir.undefined f32
+    fir.has_value %0 : f32
+}
+```
+
+This would look similar for function style `operations`.
+
+The application and access of this attribute is aided by an OpenMP Dialect 
+MLIR Interface named `DeclareTargetInterface`, which can be utilised on 
+operations to access the appropriate interface functions, e.g.:
+
+```C++
+auto declareTargetGlobal = 
+llvm::dyn_cast<mlir::omp::DeclareTargetInterface>(Op.getOperation());
+declareTargetGlobal.isDeclareTarget();
+```
+
+# Declare Target Fortran OpenMP Lowering
+
+The initial lowering of `declare target` to MLIR for both use-cases is done
+inside of the usual OpenMP lowering in flang/lib/Lower/OpenMP.cpp. However some
+direct calls to `declare target` related functions from Flang's lowering bridge 
+in flang/lib/Lower/Bridge.cpp are made.
+
+The marking of operations with the declare target attribute happens in two 
+phases, the second one optional and contingent on the first failing. The 
+initial phase happens when the declare target directive and its clauses 
+are initially processed, with the primary data gathering for the directive and
+clause happening in a function called `getDeclareTargetInfo`. This is then used
+to feed the `markDeclareTarget` function, which does the actual marking 
+utilising the `DeclareTargetInterface`. If it encounters a variable or function
+that has been marked twice over multiple directives with two differing device
+types (e.g. `host`, `nohost`), then it will swap the device type to `any`.
+
+Whenever we invoke `genFIR` on an `OpenMPDeclarativeConstruct` from the 
+lowering bridge, we are also invoking another function called 
+`gatherOpenMPDeferredDeclareTargets` which gathers information relevant to the
+application of the `declare target` attribute. This information 
+includes the symbol that it should be applied to, device type clause, 
+and capture clause, and it is stored in a vector that is part of the lowering
+bridge's instantiation of the `AbstractConverter`. It is only stored if we 
+encounter a function or variable symbol that does not have an operation 
+instantiated for it yet. This cannot happen as part of the 
+initial marking as we must store this data in the lowering bridge and we 
+only have access to the abstract version of the converter via the OpenMP 
+lowering. 
+
+The information produced by the first phase is used in the second phase, 
+which is a form of deferred processing of the `declare target` marked 
+operations that have delayed generation and cannot be proccessed in the 
+first phase. The main notable case this occurs currently is when a 
+Fortran function interface has been marked. This is 
+done via the function 
+`markOpenMPDeferredDeclareTargetFunctions` which is called from the lowering
----------------
skatrak wrote:

```suggestion
`markOpenMPDeferredDeclareTargetFunctions`, which is called from the lowering
```

https://github.com/llvm/llvm-project/pull/78502