[Openmp-commits] [PATCH] D23175: Fixes for hierarchical barrier

Terry Wilmarth via Openmp-commits openmp-commits at lists.llvm.org
Thu Aug 4 13:14:48 PDT 2016

tlwilmar created this revision.
tlwilmar added a reviewer: AndreyChurbanov.
tlwilmar added a subscriber: openmp-commits.
tlwilmar set the repository for this revision to rL LLVM.

The problem was a hang when reducing a hot team's size and then increasing it again when KMP_HOT_TEAMS_MODE=1.
Threads that were removed but kept in the team for later re-use, and then added back later, were not properly set to wait on their own b_go flag. This was properly handled for the case when threads are returned to the pool, but not when kept in the team.  However, this did not completely fix the hang.  

The second problem was that a thread that was made active in the team again still thought it knew about its related leaf threads in the hierarchy, and tried to release them by setting it's own flag, as opposed to the leaves' flags.  The leaves, however, were also recently made active again, and were thus waiting on their own b_go flags.   The fix was easy: simply reset the known leaves to zero, via the "leaf_kids" field whenever threads in a team go inactive.  The same problem would have existed for threads picked up from the pool, so this fix was made there as well.




Index: runtime/src/kmp_runtime.c
--- runtime/src/kmp_runtime.c
+++ runtime/src/kmp_runtime.c
@@ -4848,6 +4848,19 @@
             } // (__kmp_hot_teams_mode == 0)
+            else {
+                // When keeping extra threads in team, switch threads to wait on own b_go flag
+                for (f=new_nproc; f<team->t.t_nproc; ++f) {
+                    KMP_DEBUG_ASSERT(team->t.t_threads[f]);
+                    kmp_balign_t *balign = team->t.t_threads[f]->th.th_bar;
+                    for (int b=0; b<bs_last_barrier; ++b) {
+                        if (balign[b].bb.wait_flag == KMP_BARRIER_PARENT_FLAG) {
+                            balign[b].bb.wait_flag = KMP_BARRIER_SWITCH_TO_OWN_FLAG;
+                        }
+                        KMP_CHECK_UPDATE(balign[b].bb.leaf_kids, 0);
+                    }
+                }
+            }
             team->t.t_nproc =  new_nproc;
             // TODO???: team->t.t_max_active_levels = new_max_active_levels;
@@ -5331,6 +5344,7 @@
         if (balign[b].bb.wait_flag == KMP_BARRIER_PARENT_FLAG)
             balign[b].bb.wait_flag = KMP_BARRIER_SWITCH_TO_OWN_FLAG;
         balign[b].bb.team = NULL;
+        balign[b].bb.leaf_kids = 0;
     this_th->th.th_task_state = 0;

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D23175.66848.patch
Type: text/x-patch
Size: 1399 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/openmp-commits/attachments/20160804/1de5fcbe/attachment.bin>

More information about the Openmp-commits mailing list