<table border="1" cellspacing="0" cellpadding="8">
<tr>
<th>Issue</th>
<td>
<a href=https://github.com/llvm/llvm-project/issues/58311>58311</a>
</td>
</tr>
<tr>
<th>Summary</th>
<td>
OMP thread level for parallel_end callback is inconsistent.
</td>
</tr>
<tr>
<th>Labels</th>
<td>
new issue
</td>
</tr>
<tr>
<th>Assignees</th>
<td>
</td>
</tr>
<tr>
<th>Reporter</th>
<td>
SpinTensor
</td>
</tr>
</table>
<pre>
Hi all,
please consider the following test code:
```
#include <stdio.h>
#include <omp.h>
#include <omp-tools.h>
static void my_ompt_callback_parallel_begin(ompt_data_t *encountering_task_data,
const ompt_frame_t *encountering_task_frame,
ompt_data_t *parallel_data,
unsigned int requested_parallelism,
int flags, const void *codeptr_ra) {
fprintf(stderr, "Callback parallel begin called by thread %d on level %d\n",
omp_get_thread_num(), omp_get_level());
}
static void my_ompt_callback_parallel_end(ompt_data_t *parallel_data,
ompt_data_t *encountering_task_data,
int flags, const void *codeptr_ra) {
fprintf(stderr, "Callback parallel end called by thread %d on level %d\n",
omp_get_thread_num(), omp_get_level());
}
// initialize callbacks
int my_ompt_initialize(ompt_function_lookup_t lookup, int initial_device_num,
ompt_data_t *tool_data) {
// Get the set_callback function pointer
ompt_set_callback_t ompt_set_callback = (ompt_set_callback_t)lookup("ompt_set_callback");
// register the available callback functions
ompt_callback_parallel_begin_t f_ompt_callback_parallel_begin = &my_ompt_callback_parallel_begin;
ompt_set_callback(ompt_callback_parallel_begin, (ompt_callback_t)f_ompt_callback_parallel_begin);
ompt_callback_parallel_end_t f_ompt_callback_parallel_end = &my_ompt_callback_parallel_end;
ompt_set_callback(ompt_callback_parallel_end, (ompt_callback_t)f_ompt_callback_parallel_end);
return 1; // success: activates tool
}
void my_ompt_finalize(ompt_data_t *tool_data) {
(void) tool_data;
}
// start tool
ompt_start_tool_result_t *ompt_start_tool(unsigned int omp_version,
const char *runtime_version) {
static ompt_start_tool_result_t ompt_start_tool_result;
ompt_start_tool_result.initialize = &my_ompt_initialize;
ompt_start_tool_result.finalize = &my_ompt_finalize;
return &ompt_start_tool_result; // success: registers tool
}
int main(int argc, char **argv) {
fprintf(stdout, "In Main: Thread %d on level %d\n",
omp_get_thread_num(), omp_get_level());
fprintf(stdout, "\n");
for (int nthreads=1; nthreads<=3; nthreads++) {
fprintf(stdout, "Starting Parallel region with %d threads\n", nthreads);
#pragma omp parallel num_threads(nthreads)
{
fprintf(stdout, "In parallel region: Thread %d on level %d\n",
omp_get_thread_num(), omp_get_level());
}
fprintf(stdout, "\n");
}
return 0;
}
```
In the code I register callbacks for `ompt_parallel_begin` and `ompt_parallel_end`, each giving information by which thread on which level they were called. In the main function I start a parallel region with with 1,2, and 3 threads. Each of them trigger the callback functions.
I compiled the code with `$ clang -fopenmp -o test.x test.c`.
The compiler I used is:
```
$ clang --version
clang version 14.0.0
Target: x86_64-unknown-linux-gnu
Thread model: posix
```
The output looks like this:
```
In Main: Thread 0 on level 0
Starting Parallel region with 1 threads
Callback parallel begin called by thread 0 on level 0
In parallel region: Thread 0 on level 1
Callback parallel end called by thread 0 on level 1
Starting Parallel region with 2 threads
Callback parallel begin called by thread 0 on level 0
In parallel region: Thread 0 on level 1
In parallel region: Thread 1 on level 1
Callback parallel end called by thread 0 on level 0
Starting Parallel region with 3 threads
Callback parallel begin called by thread 0 on level 0
In parallel region: Thread 0 on level 1
In parallel region: Thread 1 on level 1
In parallel region: Thread 2 on level 1
Callback parallel end called by thread 0 on level 0
```
The odd thing is that the callback for the parallel end is called by thread 0 on level 1, if and only if the region was started with one thread. All other thread numbers produce reasonable and expected output.
This is a violation of the OMP standard version 5.1 section 2.6 "Parallel Construct" on page 95 where it states:
```
A thread dispatches a registered ompt_callback_parallel_end callback for each
occurrence of a parallel-end event in that thread. The callback occurs in the task that encounters
the parallel construct. This callback has the type signature
ompt_callback_parallel_end_t.
```
If I understood it correctly the thread and level issuing the callback should always be identical for the parallel begin and parallel end callback. In this case Thread 0 on level 0, which is not the case here for a region with a single thread.
</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJzVWEtz4jgQ_jXmosKFDThw4JBJZnbnMLVTNbm7hC0bbYTklWSS7K_fbsk2tnlMkp3ZR4oiYLW6v37o6xZblb9sfuWEChHEd8HsPpjd-vdKMGoYyZQ0PGea2B0jhRJCPXFZEsuMhbWcBfNGPkhmzct_jedcZqLOGQnmd8bmXIW7YP6xb2Ioo_ZVX2K8NrVKCXOiw1hqeUYOiudk_5KCoE0z8GZLs8e0oho-MpFuWcllEK_cck4tTS0J4lsmM1VLyzS4lFpqHt1aFwjyhj-MkyVOf6Hpnl0y4BbfZWGEvfPt3ZBryGwpWU64tESzP2rIKcu7oHGzf5da1FYIWhrY3YTFZQcwY8FUVqcaAK9JcPOh015UECJbQIqgVJjWuDeI47smk6QFRVwmCWYYgG9foCw1o6h8mRMliWAHEMJvwfIOMh6fcwEimZbMpn5vKmtwdAWI0Gi75hS1j9fBvIEa3Ny_pe6YzE-r7u9m7mcV8c9NHITiP5C2llw-wQsc5pZTwf9kpE2d8QIYijarR6k2lUUtM8uVTIVSj3UFafAfEAjubHakOTvwjHmg17MxSilSXZPFYbQb4L8w6_jYsGPRkRYVqRTHcug2OeV90dSePgOWvSetg0NhANH6B3GNTyRcuo6xPsLUcFaBUnzvoAfKBd2KY6w7xGYI9QJ9A-riKr83LiTfawM9oGd8WV1vIXfkRAQDdB3ZKDyXyeKaj3iAvushEs77_HNU9Vbv3KbeMWvtamZrLUkEK20xmDrLmDEwLxAKWT9QGCEIFvrohA4YteBycPJedURWqAIfH2Wu8wBwubY9LD5q-DB1KjQztbDe7mgNrA3aKPLQgWkDVf0OBvakm-2oRlMaWJ3DJNHpGzradKCLYM8vnFbHWCLs0eKo3npU-H01be7GSrqc9lQ09QJSF1GfqaOWX87WUY_IqZv98CPVZea6WxNieMGjw9WmpmrbNLXPknxBXWD64Z_tX5cxdfbOnMJCoZPOcemNQtju3aE8fr-DR_PBo_iDew1DchnBN0wW3gm-tq0e8wJBeeJ25yPUau5i0zM2cNKd33mlabmnGJPj-ADRSrs9q_72_t4h4KuZrIZo35vUH5Fah_z-FaE-k-z-3vFxmp3hvdEtzb1DNLBD45BHPh-7djcT-UJKZu5sjjpbMiMUG9N4FTsDWAHcjGY7UvIDlgiXoGpP3ZgCU-DTjsNaMwtixbjvPu6ACASYZs3UGJIGJh7n47DzuWFvOk6oLz_3FgEMV3WIdN5WY0g-IjJVoNY9sZqXZTOrnE4o4SBeEKp9xXGU7eLmix09XpBMUPB1WqiKSajhqXLX5fDZ_8tAqlH34HY7VRo8qQ12EXP5St2pnrY9wS34h80jEi3CWdhseQB2YxZr-3mVpMliWstHqZ7kVHBZP09LWbdAXAr24IlA6UoZ_jwCQY6YoSKr2k-9hgj-yCAOl3Gf0ubseLwGhXidSqKOR5zwq--Gp9auHv-eeHTJ0tnLzOnG13gV_1teXRWPfkAQ3pDa-f8kCFfF4x8Ys-EJcmcuR7ZxJArzzo7aEVMpT10DSyB5vUrxwlo4WlRSvOBn1NFmhhpPrrDfZUlJ1mgJya0QRIGwbvVCz9viKFZpldcZKqFGSXfhQ_3suWIZavLc0REgQIQXJQeuhO8LnpDJb1--onWZU5133LYMI7j0euaPwwR7YldPdzg86zqDZukyUdGSkfUSmgo2EW7dwMwustRt60jOTUVttmMIq22GCPzytWyQBex3zT0iy2qtmYRogFPHDjXFPZAB91NBm0sf1od-Tt1-42Ug8tQ8euHuF5_muAzynrVhQGVNAThtO2q8npeKEbyyUBgSWO_Gc-FGGp4n9QIblswBBAzfOcY3U-BsZsWLN-Ojibn31caNqd1Px30XzU7VAqTEE30xcMoJzyEsHNZPK9pzACo8OU6oqxkQnMuGnW01UO5-xAAhqdoTBMKuRNAgHdAShTjJUnRVP2GbKEmWy2Q1S6JJvpnn6_maTiy3gm2wYBunvT3Ud75MsOil-2UdakvacFJrsdlZW7nqdPecEuzX2xCmA_gixKH9N4Xz9TvDIv-EAYV6jj8tV_Momuw2y2JJV2yezFarOGaL-Xq1yJMVK4osWbLFejERdMuE2QRLGO9jyZ5cTpgbLO8nfBPP4jiaRfFsvbhZROFNNttuozWLcnqTJxkLFjMGw5cIEUeodDnRGwdpW5cGFgV4Y46L1Lh7MXPmQD-t7U7pzbeKywcmjdITZ33j0P8Fzm7NuA">