<table border="1" cellspacing="0" cellpadding="8">
<tr>
<th>Issue</th>
<td>
<a href=https://github.com/llvm/llvm-project/issues/63261>63261</a>
</td>
</tr>
<tr>
<th>Summary</th>
<td>
[OMPT] Testing OpenMP locks dispatch callbacks with wrong kind
</td>
</tr>
<tr>
<th>Labels</th>
<td>
new issue
</td>
</tr>
<tr>
<th>Assignees</th>
<td>
</td>
</tr>
<tr>
<th>Reporter</th>
<td>
Thyre
</td>
</tr>
</table>
<pre>
# [OMPT] Testing OpenMP locks dispatch callbacks with wrong kind
## Description
The OMPT interface offers several methods for tracking OpenMP locks. Two important ones are the callbacks `ompt_callback_mutex_acquire` and `ompt_callback_mutex_acquired` . These callbacks let us keep track if a thread tries to test or acquire an initialied lock.
We ran into an issue in [Score-P](https://www.vi-hps.org/projects/score-p/) with an OpenMP example ([simple_lock.1.c](https://github.com/OpenMP/Examples/blob/main/synchronization/sources/simple_lock.1.c)) where the method `omp_test_lock` is used to acquire a lock.
After a bit of investigation, we noticed that both `ompt_callback_mutex_acquire` and `ompt_callback_mutex_acquired` are dispatched with the incorrect `kind`. Instead of `ompt_mutex_test_lock` / `ompt_mutex_test_nest_lock`, we see `ompt_mutex_lock` / `ompt_mutex_nest_lock` when the functions `omp_test_lock` / `omp_test_nest_lock` are called.
This behaviour is non-conforming with the OpenMP specifications, where it states [[Link](https://www.openmp.org/spec-html/5.1/openmpsu184.html#x230-2630003.9.6)]:
> The above callbacks occur in the task that encounters the lock function. The kind argument of these callbacks is `ompt_mutex_test_lock` when the events arise from an `omp_test_lock` region while it is `ompt_mutex_test_nest_lock` when the events arise from an `omp_test_nest_lock` region.
In our testing, we were able to reproduce the issue with Clang 13.0.1, 15.0.6 and 16.0.3. Older versions were not tested. Other compilers based on LLVM, more specifically ROCm and Intel oneAPI, are affected as well.
Hopefully this information will help you fix the issue. If you need more information, I will be happy to help.
## Reproducer
A reproducer can be found below. Here, we register an OMPT interface with the following callbacks:
- `ompt_callback_thread_begin`
- `ompt_callback_lock_init`
- `ompt_callback_lock_destroy`
- `ompt_callback_mutex_acquire`
- `ompt_callback_mutex_acquired`
- `ompt_callback_mutex_released`
We use the `thread_begin` callback to set a unique identifier for our threads. The other callbacks are mostly present to print out our information. The important callbacks here are `ompt_callback_mutex_acquire` and `ompt_callback_mutex_acquired`.
The code includes a main function where we acquire a lock via `omp_test_lock` / `omp_test_nest_lock`, do some dummy work and release the lock afterwards. At the end of the program, we check inside of `my_finalize_tool` if we encountered `ompt_mutex_test_lock` / `ompt_mutex_test_nest_lock`. If that's not the case, we abort via an assertion.
Running the reproducer, we get the following output:
```bash
$ clang -fopenmp error_omp_test_lock.c
$ ./a.out
[ompt_start_tool] tid = -1 | omp_version 201611 | runtime_version = 'LLVM OMP version: 5.0.20140926'
[my_initialize_tool] tid = -1 | initial_device_num 0
[lock_init_cb] tid = 1 | kind = lock | wait_id = 140729574265216 | codeptr_ra = 0x40128a
[mutex_acquire_cb] tid = 1 | kind = lock | wait_id = 140729574265216 | codeptr_ra = 0x401355
[mutex_acquire_cb] tid = 2 | kind = lock | wait_id = 140729574265216 | codeptr_ra = 0x401355
[mutex_acquired_cb] tid = 1 | kind = lock | wait_id = 140729574265216 | codeptr_ra = 0x401355
[mutex_released_cb] tid = 1 | kind = lock | wait_id = 140729574265216 | codeptr_ra = 0x401387
[mutex_acquire_cb] tid = 2 | kind = lock | wait_id = 140729574265216 | codeptr_ra = 0x401355
[mutex_acquired_cb] tid = 2 | kind = lock | wait_id = 140729574265216 | codeptr_ra = 0x401355
[mutex_released_cb] tid = 2 | kind = lock | wait_id = 140729574265216 | codeptr_ra = 0x401387
[lock_destroy_cb] tid = 1 | kind = lock | wait_id = 140729574265216 | codeptr_ra = 0x4012ce
[lock_init_cb] tid = 1 | kind = nest_lock | wait_id = 140729574265208 | codeptr_ra = 0x4012d7
[mutex_acquire_cb] tid = 1 | kind = nest_lock | wait_id = 140729574265208 | codeptr_ra = 0x4013b5
[mutex_acquired_cb] tid = 1 | kind = nest_lock | wait_id = 140729574265208 | codeptr_ra = 0x4013b5
[mutex_acquire_cb] tid = 1 | kind = nest_lock | wait_id = 140729574265208 | codeptr_ra = 0x401245
[mutex_acquire_cb] tid = 2 | kind = nest_lock | wait_id = 140729574265208 | codeptr_ra = 0x4013b5
[mutex_released_cb] tid = 1 | kind = nest_lock | wait_id = 140729574265208 | codeptr_ra = 0x4013e6
[mutex_acquire_cb] tid = 2 | kind = nest_lock | wait_id = 140729574265208 | codeptr_ra = 0x4013b5
[mutex_acquired_cb] tid = 2 | kind = nest_lock | wait_id = 140729574265208 | codeptr_ra = 0x4013b5
[mutex_acquire_cb] tid = 2 | kind = nest_lock | wait_id = 140729574265208 | codeptr_ra = 0x401245
[mutex_released_cb] tid = 2 | kind = nest_lock | wait_id = 140729574265208 | codeptr_ra = 0x4013e6
[lock_destroy_cb] tid = 1 | kind = nest_lock | wait_id = 140729574265208 | codeptr_ra = 0x40131b
[my_finalize_tool] tid = 1
a.out: error_omp_test_lock.c:217: void my_finalize_tool(ompt_data_t *): Assertion `ompt_mutex_test_lock_kind_encountered == 1' failed.
Aborted
```
Source code:
```c
#include <assert.h>
#include <inttypes.h>
#include <omp.h>
#include <omp-tools.h>
#include <stdatomic.h>
#include <stdbool.h>
#include <stdio.h>
#include <unistd.h>
/********************
* Main Code
********************/
void skip() {
// Placeholder function for skipping work
sleep(1);
}
void nest_work(omp_nest_lock_t* lck)
{
omp_set_nest_lock(lck);
sleep(1);
omp_unset_nest_lock(lck);
}
int main() {
// Test simple OpenMP locks
omp_lock_t lck;
omp_init_lock(&lck);
#pragma omp parallel shared(lck) num_threads(2)
{
while (!omp_test_lock(&lck)) {
skip();
}
sleep(1);
omp_unset_lock(&lck);
}
omp_destroy_lock(&lck);
// Test nested OpenMP locks
omp_nest_lock_t nest_lck;
omp_init_nest_lock(&nest_lck);
#pragma omp parallel shared(nest_lck) num_threads(2)
{
while (!omp_test_nest_lock(&nest_lck)) {
skip();
}
nest_work(&nest_lck);
omp_unset_nest_lock(&nest_lck);
}
omp_destroy_nest_lock(&nest_lck);
return 0;
}
/********************
* OMPT Code
********************/
int32_t ompt_mutex_test_lock_kind_encountered = 0;
_Thread_local int32_t ompt_tool_tid = -1; /* thread counter. >= 1 after thread_begin */
static const char *
mutex2string(ompt_mutex_t t)
{
switch (t)
{
case ompt_mutex_lock:
return "lock";
case ompt_mutex_test_lock:
return "test_lock";
case ompt_mutex_nest_lock:
return "nest_lock";
case ompt_mutex_test_nest_lock:
return "test_nest_lock";
case ompt_mutex_critical:
return "critical";
case ompt_mutex_atomic:
return "atomic";
case ompt_mutex_ordered:
return "ordered";
}
assert(false);
return "";
}
static const char *
scope_endpoint2string(ompt_scope_endpoint_t t)
{
switch (t)
{
case ompt_scope_begin:
return "begin";
case ompt_scope_end:
return "end";
case ompt_scope_beginend:
return "beginend";
}
assert(false);
return "";
}
void thread_begin_cb(ompt_thread_t thread_type,
ompt_data_t *thread_data)
{
assert(ompt_tool_tid == -1);
static atomic_int_least32_t thread_counter = 1; // ompt_tool_tid >= 1
ompt_tool_tid = atomic_fetch_add(&thread_counter, 1);
thread_data->value = ompt_tool_tid;
}
void lock_init_cb(ompt_mutex_t kind,
unsigned int hint,
unsigned int impl,
ompt_wait_id_t wait_id,
const void *codeptr_ra)
{
printf("[%s] tid = %" PRId32 " | kind = %s | wait_id = %" PRIu64 " | codep `ompt_callback_mutetr_ra = %p\n",
__FUNCTION__,
ompt_tool_tid,
mutex2string(kind),
wait_id,
codeptr_ra);
}
void lock_destroy_cb(ompt_mutex_t kind,
ompt_wait_id_t wait_id,
const void *codeptr_ra)
{
printf("[%s] tid = %" PRId32 " | kind = %s | wait_id = %" PRIu64 " | codeptr_ra = %p\n",
__FUNCTION__,
ompt_tool_tid,
mutex2string(kind),
wait_id,
codeptr_ra);
}
void mutex_acquire_cb(ompt_mutex_t kind,
unsigned int hint,
unsigned int impl,
ompt_wait_id_t wait_id,
const void *codeptr_ra)
{
printf("[%s] tid = %" PRId32 " | kind = %s | wait_id = %" PRIu64 " | codeptr_ra = %p\n",
__FUNCTION__,
ompt_tool_tid,
mutex2string(kind),
wait_id,
codeptr_ra);
if( kind == ompt_mutex_test_lock || kind == ompt_mutex_test_nest_lock )
{
ompt_mutex_test_lock_kind_encountered = 1;
}
}
void mutex_acquired_cb(ompt_mutex_t kind,
ompt_wait_id_t wait_id,
const void *codeptr_ra)
{
printf("[%s] tid = %" PRId32 " | kind = %s | wait_id = %" PRIu64 " | codeptr_ra = %p\n",
__FUNCTION__,
ompt_tool_tid,
mutex2string(kind),
wait_id,
codeptr_ra);
if( kind == ompt_mutex_test_lock || kind == ompt_mutex_test_nest_lock )
{
ompt_mutex_test_lock_kind_encountered = 1;
}
}
void mutex_released_cb(ompt_mutex_t kind,
ompt_wait_id_t wait_id,
const void *codeptr_ra)
{
printf("[%s] tid = %" PRId32 " | kind = %s | wait_id = %" PRIu64 " | codeptr_ra = %p\n",
__FUNCTION__,
ompt_tool_tid,
mutex2string(kind),
wait_id,
codeptr_ra);
}
static int
my_initialize_tool(ompt_function_lookup_t lookup,
int initial_device_num,
ompt_data_t *tool_data)
{
printf("[%s] tid = %" PRId32 " | initial_device_num %d\n",
__FUNCTION__,
ompt_tool_tid,
initial_device_num);
ompt_set_callback_t set_callback =
(ompt_set_callback_t)lookup("ompt_set_callback");
set_callback(ompt_callback_thread_begin, (ompt_callback_t)&thread_begin_cb);
set_callback(ompt_callback_lock_init, (ompt_callback_t)&lock_init_cb);
set_callback(ompt_callback_lock_destroy, (ompt_callback_t)&lock_destroy_cb);
set_callback(ompt_callback_mutex_acquire, (ompt_callback_t)&mutex_acquire_cb);
set_callback(ompt_callback_mutex_acquired, (ompt_callback_t)&mutex_acquired_cb);
set_callback(ompt_callback_mutex_released, (ompt_callback_t)&mutex_released_cb);
return 1; /* non-zero indicates success */
}
static void
my_finalize_tool(ompt_data_t *tool_data)
{
printf("[%s] tid = %" PRId32 "\n",
__FUNCTION__,
ompt_tool_tid);
assert( ompt_mutex_test_lock_kind_encountered == 1 );
}
ompt_start_tool_result_t *
ompt_start_tool(unsigned int omp_version,
const char *runtime_version)
{
setbuf(stdout, NULL);
printf("[%s] tid = %" PRId32 " | omp_version %d | runtime_version = \'%s\'\n",
__FUNCTION__,
ompt_tool_tid,
omp_version,
runtime_version);
static ompt_start_tool_result_t tool = {&my_initialize_tool,
&my_finalize_tool,
ompt_data_none};
return &tool;
}
```
</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJzsW1tv27iz_zTMyyCCTEW-POQhlxr_AO226GbPeRRoibZ4IpFakoqb_fQHQ8m6Wb4kTXYL_DcI2lgczpAzv_lxSNHMGLGRnF-T8JaE9xestKnS14_pi-YXK5W8XBMaAAlvv3759kjCe3jkxgq5ga8Fl1--QabiJwOJMAWzcQoxy7IVw0dbYVPYaiU38CRkQvx74t_U_9IAld5zE2tRWKFkt_Ux5YDGQEjL9ZrFHNR6zbUBw5-5Zhnk3KYqMbBWGqxm8dNwPB48bhWIvFDaMmlBSW6AaQ425Z0hkqmv8sJGuydRXlr-I2Lxn6XQnEx9YDI5IZWgmAePKTdd1Rm3UBp44ryohghiDQxsqjlLwGrBDVgFlhsLSkOtDJgEIYUVLBM8cXPxoOub_-WgnYxVTtaYkoOQGJ_fY6X55TcS3hM6T60tDAluCF0Sutxut96zuEwL4ym9IXRZaPV_PLaG0KVx3QonuKiCxuTOmfwHy4uMA6FzEt4agR8iN6qJF49Z2giblisvVjmhy0oJoctPlRo0t8rUitBlzoRE4y8yTrWS4i_mUECXRpU6dpJDa3ThBpjyOowVCOroROhIJ4vhEAZKwxN0cOPYnjNv1pZrYLASFtQahHxGUG_qQdzBloNUVsSoI2UWVsqm74cWBOIuYXhS-RxnJGSstOaxRSUuZ6a-Bw_SWMSMWje6K5W9KRO6HGuWHZl6YobzgeRhHd3u6HrpxrkuZYyeMqPOb7XsD8FNHT3DE6-f8sLAiqfsWahSY_ykkpexkmulc0zuxkc1ME3BY7EWsYuYcTNzwBAWjGWWG3B0dvtZyKdDGaEKLvOizghUeJnaPCN0GXoTQpdVsykn8yuvagh-0MC_pNPA9_3AW3hTxGR4j0qruQSfkAeArdRzlwtUHOOkKudZZp4qUHEZqxIpzrgGdFHjWkcojjeB6U2Zc-mAagcsI8wxTDQB489cWiRAYTistcoxxcdip_lGKAnbVGTOl-P6x1Fx0ojct9QDwYMEDL6t1pcarVuMKltlHJNZ80KrpIwrAqjIzyHjLmNyA5PA8zF0dzAJPd-bunycTD3fCzz4miVcwzPXxiHX6ZXKOnM88eCrTbmGWOWFyDAkK4YMoiR8_vw_X1BnrjRvcZdlL_D9613ubDxIyzNcZG6-PaAoopyt1zy2PAGGxrKsz-P_UQVfl6jEIvSFA7oDM2xFlkHKswJeVAlr8aOdrAcPa_dUcp5UA-r0RMsPVfcVh5QVxQs6DVV5I6vv950zdbfxpnWyhphJVLVWpUxgxTO19eA_XPM6NhhD44hUDhfsJl_XKsvUFlO4QW2TL5f7ZFmtj9GKb4REyjokhzCKcKU8KZRwY7V6OSY35PHzBJPTkppnHGHUSDaLeGkqDJOpP5hy4yeMneEWGJRS_InLfMKlFWvBtSt8XK64vqZiC1UhuCEHRGGujM1eoNDcIIVYBYUWyCWldQo68KmUtDVTq8gxK2p7rxXQG5Z7sUrc8peVCRZqgOVBw4U1tW_5YDGHZ8FevQIhdBMFRuUckjLPX2Cr9JMbdx2tlowZVglbptHBN7YiOZnUPAyFVhvN8joX4pRjjSeNSHi9VOcv0VpIlom_eGSVylxlskbhhvl58pNrumMEXE0InZmKz1yFa3Y5ylZKW-cpJoEZw3UV624EvpdSYoZi1zb7awUbbgeJrEpblLZd9aZ-9btiJt0xzBXEjpIv19UyClxrpaNeqLwYWnGP0CXzEJb1s_DWzdtYpm3lvvAerEiABPdwOQEyuwNUVzM6UH8ynVSPdSmtyHnThD0InSGRI0_tFgES3ACuE9SfXPkLOiV01tjOX6JdHb6L3r75WiJK-LOIeSTLHPxGQ8NQUbzq9q26upUdPzqg4ZMtEzbayVz5M7oIZ1d0GtLJ1LVjihRWR5o5Ef_HlT-hc9aOuJtkH2gzCMOzbNK_w2byt050R-cfaXQ--3W9-7FGD3j3XY12vNstDj4yRWP-Wkpo-P24UX9-2GhyHore3XCweluifqDlD_f11Zv48CNmfB4_vYNlPv1VpnweU_198Hp3w_vwOo8n3zfK55PlO9idrLplWL-E7hqupFzRiKXcaIVJghs6mWHzsxIJ7Omjc1dlJsyyyAKhN4QuUPpmVykfqs8jnHHUK-SD-2pYdAZrJtojrhsswXkyqJa7Bfjv7tDTeWO_rI6bXXu9QQIS3FWVvJeS4NNYs5DWvhTcHBRQeXGs7RKdc7i3sQmzKhfxMYmVUtmxdqEOtpZSGJt0m2uhpYvQW3-dEgwyfMH95R26u3n2E7_L7iAdzsyTKAidE7oAMrutjQBAdfgI3zIW81S5E6lml4t7euxXuPNOpZ_abibjHPVNHDprfWR2v2fXZZ_r64DdbhUji9PO4ifUUHfvjAtlDe9uLem8Ft6ZGx_ErnMpT3QfjLb6V0gL1ZuAgatqPz1yY6F6D9B7vdO3Xc3PTa4ZFj53lVY9GkKnwwHtcFdotskZ9oCCaZZlPAOTMs2TZhYgy7w-mjKEzmnjRBfTriPxpzo9dUYn_bOJ7jh6823x0vNspf6-_2AcDP6iDcPBOcMgCjsP7pj9hLM6CHaRke7o9EhkOvCrF4bRGMm-gxrJV0er0_MdQnZkWMO8bmJzThy7SXpwuovxrDog3zPQce8usGd7GBVpbkstwT-Yu-9Gw-7Q-GNoWEgb0MjC2Wt3Z77RY3Ukm6mYZdDThOti1B4DkeC2yoib3ZvdWqcHuHa58sidIkL3lBf2h2sssyKGWEljIU6ZhsZTbvDUWO1eisy7EwI7QudmK2ycIpjtYdDHzHAYvgIMbvqQrYFAKK2gQ3t4G6poae6wng4VHlcmz1Amz1U2yOYTwztbbayFFTHLjihsRE6oqqqpI4pqgRNqlE4Qzkf07CQGinqLTFVbEjpfs8zwIc20urpKBhRxBM4mVgWPuEwKJaQd4LrfeADgyLOvw3iltnrDctg3VXvXM0MNXB7zLbYeDFBnCMe1NCIHQ_QO8XHFYpeScFNXx6B-bHftuJEg9K5dVbrbpVoGH4xHqhnsHn3WDNorMSvYVGCPEAG41a3ot7ZU82u9A9zR73KPnT91togj1F2bWHMbpxFLkmpd7NtwL5J74-vM9pIEn55ZVnKnrmfgqNN7Z4ADNndXPhpP7_2U0t3USnBFglRI28r2mrBePqLGmaw35pHdbdGPdKiy2I2e0Jt2yz4ScPdyce2cSUl4S2hoult2QkNCKXz7_pAEFCHaPztA8b1jg7ZPOb1q-rhRjL9pbI8TCA0LEt65nB6ZXhQt__jt7vHh629RNAB4E8qRboP1uAraopU84tC-707CpHPg8jqkvCXKv3Kg_-mQtj97fjw_pHvHhkdjeiTZR39ewwDn4OMVWAD4teAwDoLB1Pfh8BMgOJXhINA3zYSbFWNQOOMMu44Zk-ucrzaFz95W9PxNz2TsbOAUeJOT6H0L-7yKf_7F3H8L5rqvO_7F3D-LubMWu7qIxyWrOr_Yv0pTx3F39B1lSj2VRWSh-qM16BazvVs2x2I63J_g9A_vTt4Uz5FrP4SGyfuXJWMz3zt6t-7gvr1ACd2PLqPrVNttsXvShC52Pkcf7Em4Ce0dEvYl5keucNI72BdwcJvubT-7G8FjBtq7n0e09_dZr1K9uzF6Snu3PD_XQP_C5BEL-9Xim0wkZ9tI3mCkud960kiPxY8cOndPVKWSl39xrUDIRMTuUr8p45gb0z9AHaUfZPSGf068831nkvhZou6neHOAcv4KW7-WP0zRg5uVkeamzGz9AnxMgtB5b4PRuXh5kIz7h3-DO5lj59bcrkp0s7GJKl1y__bH589Df7yJsrsXRZGrD18TDe8InVVK3V_vTuqjrhvxTnfONaYPhg0_VcOf3WLKjay4d80yMN1PiP3DPakkR9CMnixOXa99bO1uLFwk10GyCBbsgl9PpvNpGM5mIb1IryfJNEhCf3Y1D32WLELqc8bCcBXyMKBTzi7ENfVp4E8n1J9PwpB6IV2wKWNTxmkQBnRBrnyeM5F5Wface0pvLtwXI66nAZ1OLjK24plx32SkVPJt9a0JB5T7C32NfS5X5caQKz8TxppWixU2c1-B_MlvOV6UOrs-8pU4NFj_d1l_CY_QpRumIXTppvH_AQAA___1ekqH">