[llvm] [offload] Extract OMPT out of PluginInterface (PR #221726)

[Alex Duran via llvm-commits]

adurang wrote:

> Please do correct me if I am making assumptions about goals/architecture here as I have not been around for some of the discussion.
> 
> > In my quick look, I feel that some of the operations after the init but before returning to libomptarget my trigger other OMPT events (they would happen before the init in trace which would be definitely weird).
> 
> This patch removed all the remaining OMPT bits from liboffload. This means that libomptarget is now in full control of doing the callbacks. As long as we only call liboffload APIs to init the device, doing the callback immediately after those liboffload API calls should guarantee that we do not accidentally emit weird callbacks before the init callback fires.
> 
> > Also, when we move to liboffload there's no direct control to initialize/deinit devices (and afaik no intention to add them) so the placement of these event would become "best-guesses" which could lead to all kind of trace consistency issues.
> 
> Regarding this, the OpenMP standard states this:
> 
> ```
> The OpenMP implementation dispatches this callback after OpenMP is
> initialized for the device but before execution of any construct is started on the device.
> ```
> 
> ```
> The OpenMP implementation dispatches this callback immediately prior to finalizing the
> device that the device_num argument identifies.
> ```
> 
> The state after this patch satisfies these. As you said we need to be careful not to fire other callbacks before init and after finalize. I think moving OMPT back to libomptarget and giving it full control over the callbacks makes this easier compared to liboffload being able to do the callbacks.
> 
> _Thoughts on more overarching future direction below:_
> 
> _If_ we decide having callbacks in liboffload is a good idea I think it will need significantly more infrastructure than now to satisfy the correctness requirements in the presence of multiple offloading languages in the same executable.
> 
> I think in that case specifically for device init/deinit the best idea is to decouple liboffload's notion of "device is not initialized" from each user of liboffload's notion of it.
> 
> For example if OpenACC and OpenMP coexist, they will both link to the same liboffload.so, so they will share the same memory state of that library. If OpenMP happens to do its init first, then init will fire for OpenMP but not for OpenACC if we leave it to liboffload to fire callbacks for us. So I think moving those events to libomptarget/libacctarget is the best option for now and that is already what happens for the rest of the OMPT callbacks.
> 
> (there is some light discussion of this [here](https://discourse.llvm.org/t/rfc-openacc-runtime-in-llvm-offload-libacctarget/90793))
> 
> Eventually, I think a callback implementation that fires events from liboffload will need to carry along some 'listener type' in the function call stack so that the appropriate events can be fired when liboffload executes some action.
> 
> For example, for calls into offload that came from OpenACC libacctarget will attach its own listener to the call, and then whenever liboffload does some action it will notify that listener and then libacctarget can appropriately route it to the acc callback, and analogous for OpenMP. This would give each user of liboffload the ability to customize what callbacks get emitted for each call into the liboffload APIs (which can themselves have complex calls within liboffload which will need to carry around the "listener" for the current call stack)
> 
> > I feel like @jplehr changes to fix OMPT in the plugins are a better solution (although far more complex). Do you think it might be possible just to take out the OMPT user-ids for now somehow as a first step?
> 
> I am not aware of these changes, do you have pointers for this?

Ok, I think you might be correct. Have you done any testing for OMPT?

And yes, the direction that I discussed with @jplehr is exactly as you say: pass extra info on the calls that allows to send the tracing information to the appropriate layer (not just a fixed one like now). 

https://github.com/llvm/llvm-project/pull/221726