<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/65104>65104</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
            [OMPT] Overlapping `device_num` when using multiple offloading architectures
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
            new issue
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          Thyre
      </td>
    </tr>
</table>

<pre>
    ## Description

This issue was initially found during discussion of #64487. However, this should be handled as a separate issue, especially since the other one is closed now. This issue doesn't affect me directly, but there are users which might run into this issue, especially in mixed GPU systems. 

When a user utilizes multiple architectures to offload to in his program (for example `-fopenmp-targets=x86_64,nvptx64`, the OMPT interface will receive callbacks for `ompt_callback_device_initialize` for each device used. While this works fine, we may encounter a case where the transferred device number for two different architectures overlap, causing issues for tools to differentiate the devices. 

Users can select their device by the device number, for example
```c
int main( void )
{
// x86_64 (four devices)
#pragma omp target device(0)
{...}
// nvptx64 (other devices)
#pragma omp target device(4)
{...}
}
```

This would translate to the following in the OMPT interface 
```console
Callback Init: device_num=0 type=generic-64bit
[...]
Callback Init: device_num=0 type=sm_80 
```

## Reproducer

This small code should show the issue. I have tested it with with `nvptx64 + x86_64` and `nvptx64 + amdgcn-amd-amdhsa`, but it should also affect other combinations

```c
#include <assert.h>
#include <stdio.h>
#include <omp-tools.h>


void callback_ompt_device_initialize(int device_num,
 const char *type,
 ompt_device_t *device,
 ompt_function_lookup_t lookup,
                                     const char *documentation)
{
    printf("[%s] device_num = %d | type = %s\n", __FUNCTION__, device_num, type);
}

static int
initialize_tool( ompt_function_lookup_t lookup,
 int                    initialDeviceNum,
                 ompt_data_t* toolData )
{
    ompt_set_callback_t set_callback =
        ( ompt_set_callback_t )lookup( "ompt_set_callback" );
    assert( set_callback != 0 );

    ompt_set_result_t registration_result = set_callback(ompt_callback_device_initialize, (ompt_callback_t) &callback_ompt_device_initialize);
    assert(registration_result == ompt_set_always);

    return 1;
}

static void
finalize_tool( ompt_data_t* toolData )
{
}
        
ompt_start_tool_result_t*
ompt_start_tool( unsigned int omp_version, /* == _OPENMP */
                 const char* runtime_version )
{
    static ompt_start_tool_result_t tool = { &initialize_tool,
 &finalize_tool,
 ompt_data_none };
    return &tool;
}


int main(void) {
 #pragma omp target device(0)
    {}

    #pragma omp target device(4)
    {}
}
```

Running the tool on a system with a single NVIDIA MX550, we can see the following output:
```console
$ clang --version
clang version 18.0.0 (https://github.com/llvm/llvm-project.git 4b383107fa7585bb5ecd7f03cab7800b33d1585a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/software/software/LLVM/git/bin
$ clang -fopenmp -fopenmp-targets=x86_64,nvptx64 reproducer.c
$ ./a.out
[callback_ompt_device_initialize] device_num = 0 | type = generic-64bit
[callback_ompt_device_initialize] device_num = 0 | type = sm_75
```

## Previous discussion

Here are a few related comments by @mhalk and me regarding this issue:

https://github.com/llvm/llvm-project/issues/64487#issuecomment-1673631414
> Let's say we would go with "added support": I added prints for `omp_get_device_num` and `omp_get_num_devices`. Also forced the target regions to use device number 3 (and 11, for the two-target-version).
> 
> Output for `x86_64` only:
> 
> ```
> trunk/bin/clang -fopenmp -fopenmp-targets=x86_64 veccopy-ompt-target.c -o veccopy-ompt-target
> ./veccopy-ompt-target
> 
> omp_get_device_num=4
> omp_get_num_devices=4
> 
> Callback Init: device_num=3 type=unknown device=0x1485000 lookup=0x7f0bd39c8a90 doc=(nil)
> Callback Load: device_num:3 module_id:0 filename:(null) host_adddr:0x200368 device_addr:(nil) bytes:17112
> [...]
> Callback Fini: device_num=3
> ```
> 
> Output for `x86_64` and `amdgcn-amd-amdhsa` (8 gfx90a GPUs present):
> 
> ```
> trunk/bin/clang -fopenmp -fopenmp-targets=x86_64,amdgcn-amd-amdhsa veccopy-ompt-target.c -o veccopy-ompt-target
> ./veccopy-ompt-target
> 
> omp_get_device_num=12
> omp_get_num_devices=12
> 
> Callback Init: device_num=3 type=gfx90a device=0x12ae980 lookup=0x7f441a241a90 doc=(nil)
> Callback Load: device_num:3 module_id:0 filename:(null) host_adddr:0x200378 device_addr:(nil) bytes:19208
> [...]
> Callback Init: device_num=3 type=unknown device=0x12b52a0 lookup=0x7f441a241a90 doc=(nil)
> Callback Load: device_num:3 module_id:0 filename:(null) host_adddr:0x204f00 device_addr:(nil) bytes:17112
> [...]
> Callback Fini: device_num=3
> Callback Fini: device_num=3
> ```
> 
> So, in the sources, I intentionally used devices 3 and 11 for target regions. Which both correlate to device number 3 in their respective RTL. That is, to illustrate that there might be ambiguous output w.r.t. `Fini` callbacks.


https://github.com/llvm/llvm-project/issues/64487#issuecomment-1674537696
> > Judging by your output for both `x86_64` and a combination of `x86_64` + `amdgcn`, I would expect that `nvptx64` + `amdgcn` yields the same results shown in your second example. I _think_ that I can test this. For tool developers, the ambiguous output can be hard to work with, since we can only use the `device_num` to identify the executing device during target callbacks. The runtime should probably dispatch the same device number users can use to define the executing device (here `3` for the AMD GPU and `11` for the host).
> 
> I checked how the device numbers are delivered on a system with an AMD + NVIDIA GPU. We can see the same behavior:
> 
> ```
> $ clang-18 -fopenmp -fopenmp-targets=nvptx64,amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx90a veccopy.c
> $ ./a.out
> Num devices = 3
> Device 0
> Callback Init: device_num=0 type=sm_80 device=0x556d3f652a90 lookup=0x7efd6a8027b0 doc=(nil)
> Callback Load: device_num:0 filename:(null) host_adddr:0x556d3ed12778 device_addr:(nil) bytes:715888
> [...]
> Device 1
> Callback Init: device_num=0 type=gfx90a device=0x556d402203d0 lookup=0x7efd6a8027b0 doc=(nil)
> Callback Load: device_num:0 filename:(null) host_adddr:0x556d3edc1478 device_addr:(nil) bytes:25064
> [...]
> Device 2
> Callback Init: device_num=1 type=gfx90a device=0x556d40229210 lookup=0x7efd6a8027b0 doc=(nil)
> Callback Load: device_num:1 filename:(null) host_adddr:0x556d3edc1478 device_addr:(nil) bytes:25064
> [...]
> Success
> Callback Fini: device_num=0
> Callback Fini: device_num=0
> Callback Fini: device_num=1
> ```




https://github.com/llvm/llvm-project/issues/64487#issuecomment-1674568211
> > The runtime should probably dispatch the same device number users can use to define the executing device [...]
> 
> tl;dr: Agreed. Like that idea, albeit I'd have to see if this is possible. Maybe I'm not aware of some information but ATM I'm not very confident this can be (reasonably) solved on my end.
> 
> When the callbacks are dispatched, we only have the information from the corresponding RTL. To adhere to our example, that is e.g. `DeviceId=3` -- while in the OpenMP runtime this might in fact be device number 11. Now the callback is executed, that's it -- for the init callback we could (during execution of the corresponding callback!) find out the "actual" (whatever that means) device number since we have the `kind` (amdgcn, nvptx, ...) and might be able to deduce further info. But other callbacks do not have this information.
> 
</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJzMWltz2ziy_jX0S5dVJKgL9eAH24pnfSpOXLPOzr6pQKIp4pgEVABoWfvrTzVAStTFiXJmdrOuGYcmbo3ur7u_BsitlSuFeBNN7qLJ4oq3rtLm5qXaGrzKtdjeRCyNWAoLtIWRaye1iuJFFN-G3y-VtCCtbRE23IJU0kle11sodasEiNZItQIhbdFaK7UCXULE0ul4nM1G8De9wTc0EbsHRxPZSre1gByh4krUKIBb4GBxzQ13GBai3mjXWISFrFQFgqsQtKvQgFbUD4paWxSg9GYEAyGFRqsiNnPAyxILBw2CkAYLV29p4rx1NJdB4AahtWgsbCpZVNDIVeXAtAqkcjrIe04eqaCR7yjgt-dvYLfWYWNHMNTZHxUq4H5yaJ2s5b_QQtPWTq5rWreopMPCtQYtOA26LGvNBT1KBbTs2uiV4Q1ELCu1AXznDY2MpvF1qdeomvW142aFzkbp4j2bLqfjiN2rt7V7n46jaRz0jfD16fmFdoOm5AXCRtY1GCxQviEUvK5zXrxaoCWiaaybtVv2b5cC32SBy87e8l8YTWPfE3lRQWilHYoR_FHJGoO-NtrQhFJ5pW0QGr4FVIVuSQjgUHCLsPH6JwGd4cqWaAyKfk7VNjkav5TbaBCyLNGgckd6029oar6mZQreWgKhN1bYjtO69rrdDZcEL1oyLHNksW8eBwVXYLEm1LgKpelFyreDkZ2AtPDANt1k0zj8V4S_pXLQcKkilsGblgIiNu96zu66B_YQsQcIRgwGb_uF7b47S9eGrxoOullDsH3XKWJZPJx2NBpFs8XB5B0waPbgQj83_fjD6XcP_b5PIsfG-7s3c-0toL0qS13XeuONps4h9VidWlndK_m-gyg8Kumi9LYTdKnaJkoXMbjtGqN0sUKFRhbX03EuXTffxIs_WfzURLZZZvGxRMONdgH0d1wbLdoCzYkWbMPrGgotsI-AttIbv3EP2hE8QsXfEBxahwKkg410VfgVTeO9Ae86pJA3ciWOG3kjVoW65o2g_yvLu2BAUU-6fnFeW92HxwCIQje5VJyCvz3Y2hGgI5ZKVdStQIjSe24tGjeqovTTuWbrhNQftmoKYuSmwx6D395fduHIB6fTmMQycrGB3dh9GA0EGQdFxQ1E7Nabctc2nMxRcw_1gw5lqwpSyLLW-rVdLx2Eh323S34O5RC6aBtUzmv6NBjQgLWRypURyyLGKGeziY0mi8EeIUoXELGJgGh270Hav7HR5F7RMHYPy-XDty_3L49fvyyX9PeBjgK02TxK74492f-2JGBB_tjHsV7jSzIZhbMLVUTmOfPTTbjwQn0Z2u34J9iKO750Ebv1gX3BHT8TSXe9LQ7ymIPhn6Spw4V2ezkaFbF5v5cMIsZO-kSMwVCFNFlwCBpxuChLyELxQf8zQhu0be2WDgyupHXGo6R76218uH72o4zN7uGkl4vYHCI2_aFnfbCzD0Qj6Xb74PWGb-35zRp0rVGQ_AB65P3hRSnVGeRdhIjB1F1TENFx4_x0O41H7PZsB1qvVZ5BC49l3ayXb2isd19S7wPJ0Clg-fX505enZ3J0ajgP6H1AoJGmVU422M_5Aaw7pXwkvddBCAKzOzLuib_23hWx6ZE-D2MiaVURwSbVDe3fmS1iUz_sA-OdsB5vRgLcbjeXMhnvm7O7owWCx15GVk5m-C5b-b1ViviIZ6WkTk0UPtD7kIc5VSKrGuHLPx4Xj7fw9M_JJO5obmCOeMRtdOvWLTGL79GZiI2hqLlawfV1jyzfEF72wEiyUTyiCJJVzq0tTerJ3Uq6qs1HhW4i9lDXb_0_12uj_xcLN1pJB-M8zdIknpV8NskmeT7BQszKOC14PsviOE9TkUyyCd_p7sVrlThRoBvXrXpVeqOua6na9-uVant2Y5ALaLTAmnqvtZXvoelRWcfrGsVCGmqK2INeu4g9WF26DTd4-Pj58z-ewnYi9pBLdaKcrvCBCyogMDsqNir2E40i9sBHut2zwR8FwdO8Gx8m3fMk809Pa5vlbPJjxvls8E3q1g7K72Gvv_VVLocSN2CQGLggrkcUxFJZE43jpuL1q-eSDVLi4UYEP9iVv-mBd_8c_CL2EMqyiD34MwEigfSik-I6mc7SaZqMk3G3SPoJPqOL2MyC5VtyrlBDrHRHhxnjQqAA267XmnISI3g9Qnjr6dOwpl2ucGcDYj974tw3qrZZ9hXRNB7BLdHjUpsCRQgHIcRQ5tPKV5WtPaoGISXPpHmTpK8M_dCN7nC68202H-13un_66mNFL_ee5GtVb_cWOBhyDI30EzjTqtfOgdjDpZ4Db1gUer29Jrx2raMCrvW5hv1i5E_f7bB_OmOHdDE-bR-a4qDD_un7VVvaV21dwOoTQ7qI35NxNonjuOeo9GpWxrlI50XG5zEIXVAiZ5mS9T4PD5f8rLk4XvI2pfDX1riU1BZDKWtUvPF-wzLV1jQZVNq6JRdCUDCM31kcp9Osn4iH17ulId860sBtMksSNtDCYQF7INuDVPJUHd9Fy4_R13nKubKSAJ_Bqnyfxxx-e_5mYW3QonKe9_17ARux-xORfhGIh-Y5j-IDA_4sjDv9DlHMOM6zIxSPxwln4-RXoHh2AYrnLM4uQ_HPOzXLJ4z_t6hjXMbxf9Kp_xrv_7umnNWdxFndGn82eA-P_kROUannT75buzumtZBCSHch1x2kSH8kXFSQa1dBoU1gHv449ihphjWlAeMP2J18Q_j95fMIXiruQHopnAZZ160vOolh8_4APxzY5wi8yeWqJSIUGDdsRmbkRrRpr5dpvD_tHp3WK389oxlP0tl0Ph3oOv0E_9OKFTGrfAtb3ZpeVlKfV9RJ4OXDYzl_rzLsEbG7fWjujvkeO66E7-twiM3d4ITw3CjYSqyFDZbnngFSTelvajaK7ONltVhoJfrD7hE8wtJVUr0uwxqPvgByaJ3njSN46M7gyeBY6zUa299JnBiLhvoLIeOvQDbavHqmRwPC3U9XYBEV8tSLpomm8SGpI5gIAmsZTuvxHYvW-cupALruqqqD6h4Q8FJhX4f3J6Rro3Oe11ti1mvuimqvoEMIt7uLAy8YIbyUCs9LQMUbATeaxml_nUIdb58W_jKpy7ZJMmykGPMRaXyEosLiFQX0x8kH0lnP_gXW8g0NijMlrfJrEyi6ova3528j-OOwoPXbzrHib1Kby1P7rna7TrLv5vQenueS-vU_D7pH6eJMn4abotonyy6d70q_Tpbj8i_9BF_aZhfPqO4aBMxwNAnxpTnq6LpgkKEmk6lIy-mEUToapiksxZRnMZvl_-80dWFi8iKgSNjsgmw9SyZZ9v103Skn-WnlnNIZEm0cMxan4hdrp0jGF2iHTeLp-BLlsEuVk_xYOXOW_OXKSX6Jcv7eFgVaeymDOed-f65j8nHM-uD3v4MeTDOWJIf04D-XhU6tMqjH6ii989aF25VBFCP4LF872iUFcsrKvM5ROniM2Ex095baZwtZ9udGsNbWypy4whPf5ug7N6C0A76hrKRLsLpBkKrUpgkUJ28d3L48Dfq-odlCoVXpU3uYvOML_i6CW61IQwRCq-u3kOSaLaASZzOm_zSD1LL__MHnyE7BKLoDXc81wtaqQyFLo5swAxFbu9bKn5cF1qqBi_BxgwbiTf3XAZ77eEYLOFp5ZhrCxKPwPH0aw_U1bPyHFP2N-BrV0_MOEn7rgfFKBSUvPPM9hEGSjOBLRwR2l0-0pIdA2BuJ4Q_XpKMle5YhldxzIk-4PAIjlnW0qYNRIKKn29_fSiVki1IqQfwuUDXGeOFaXocrs2xTcYdvaIJKGuTK0pjDveyY384I0TR-lUp0Rw8dg2X34esGeiBUs3k4ytyVBnndOYRoC4SyNf6-mww6grt2d_-9Q4PQHnjdqoTlve0HiLoSN6mYp3N-hTfJdJ6m2SRl7Kq6GYs0m81mcZnHOYpMlEWc8CmbZzgvRMymV_KGxSyNszSOs3E2ZqNJjiyZ51xk84yhmEXjGBsu6xGFlZE2qysfP26mkyQeX9U8x9r6r7kYU7jpP1Fi0WRxZW58KMrblY3GcS2ts_tZnHS1_wzs69PzSzRZwNfwCc2a7HdCpzfkKOG7mt2XS92HSvTu4GOcq9bUN38iTNLG_i8AAP__S20Wyw">