[Openmp-commits] [PATCH] D23175: Fixes for hierarchical barrier

[Terry Wilmarth via Openmp-commits]

tlwilmar created this revision.
tlwilmar added a reviewer: AndreyChurbanov.
tlwilmar added a subscriber: openmp-commits.
tlwilmar set the repository for this revision to rL LLVM.

The problem was a hang when reducing a hot team's size and then increasing it again when KMP_HOT_TEAMS_MODE=1.
Threads that were removed but kept in the team for later re-use, and then added back later, were not properly set to wait on their own b_go flag. This was properly handled for the case when threads are returned to the pool, but not when kept in the team.  However, this did not completely fix the hang.  

The second problem was that a thread that was made active in the team again still thought it knew about its related leaf threads in the hierarchy, and tried to release them by setting it's own flag, as opposed to the leaves' flags.  The leaves, however, were also recently made active again, and were thus waiting on their own b_go flags.   The fix was easy: simply reset the known leaves to zero, via the "leaf_kids" field whenever threads in a team go inactive.  The same problem would have existed for threads picked up from the pool, so this fix was made there as well.

Repository:
  rL LLVM

https://reviews.llvm.org/D23175

Files:
  runtime/src/kmp_runtime.c

Index: runtime/src/kmp_runtime.c
===================================================================
--- runtime/src/kmp_runtime.c
+++ runtime/src/kmp_runtime.c
@@ -4848,6 +4848,19 @@
                 }
 #if KMP_NESTED_HOT_TEAMS
             } // (__kmp_hot_teams_mode == 0)
+            else {
+                // When keeping extra threads in team, switch threads to wait on own b_go flag
+                for (f=new_nproc; f<team->t.t_nproc; ++f) {
+                    KMP_DEBUG_ASSERT(team->t.t_threads[f]);
+                    kmp_balign_t *balign = team->t.t_threads[f]->th.th_bar;
+                    for (int b=0; b<bs_last_barrier; ++b) {
+                        if (balign[b].bb.wait_flag == KMP_BARRIER_PARENT_FLAG) {
+                            balign[b].bb.wait_flag = KMP_BARRIER_SWITCH_TO_OWN_FLAG;
+                        }
+                        KMP_CHECK_UPDATE(balign[b].bb.leaf_kids, 0);
+                    }
+                }
+            }
 #endif // KMP_NESTED_HOT_TEAMS
             team->t.t_nproc =  new_nproc;
             // TODO???: team->t.t_max_active_levels = new_max_active_levels;
@@ -5331,6 +5344,7 @@
         if (balign[b].bb.wait_flag == KMP_BARRIER_PARENT_FLAG)
             balign[b].bb.wait_flag = KMP_BARRIER_SWITCH_TO_OWN_FLAG;
         balign[b].bb.team = NULL;
+        balign[b].bb.leaf_kids = 0;
     }
     this_th->th.th_task_state = 0;
 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D23175.66848.patch
Type: text/x-patch
Size: 1399 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/openmp-commits/attachments/20160804/1de5fcbe/attachment.bin>