[Openmp-dev] Hierarchy generation patch

Wilmarth, Terry L terry.l.wilmarth at intel.com
Mon Apr 13 11:27:08 PDT 2015

Hi Hal,
Some comments below.

From: Hal Finkel [mailto:hfinkel at anl.gov]
Sent: Sunday, April 12, 2015 10:49 AM
To: Wilmarth, Terry L
Cc: openmp-dev at dcs-maillist2.engr.illinois.edu
Subject: Re: [Openmp-dev] Hierarchy generation patch

Hi Terry,

-    bool uninitialized;
+    volatile kmp_int8 uninitialized; // 0=initialized, 1=uninitialized, 2=initialization in progress

Can you please define these in an enum so we can use names instead of the numbers directly (even if the variable itself is a volatile int). I'd rather that than having to remember what the values mean.
[Terry L Wilmarth] I’m pretty actively refactoring code in this area.  Can this happen in a later change?

+    numPerLevel[0] = 4;
+    numPerLevel[1] = nprocs/4;
+    if (nprocs%4) numPerLevel[1]++;
+    for (int i=noaff_maxLevels-1; i>=0; --i) // count non-empty levels to get depth
+        if (numPerLevel[i] != 1 || noaff_depth > 1) // only count one top-level '1'
+            noaff_depth++;
+    kmp_uint32 branch = 4;
+    if (numPerLevel[0] == 1) branch = nprocs/4;
+    if (branch<4) branch=4;
+    for (kmp_uint32 d=0; d<noaff_depth-1; ++d) { // optimize hierarchy width
+        while (numPerLevel[d] > branch || (d==0 && numPerLevel[d]>4)) { // max 4 on level 0!
+            if (numPerLevel[d] & 1) numPerLevel[d]++;
+            numPerLevel[d] = numPerLevel[d] >> 1;
+            if (numPerLevel[d+1] == 1) noaff_depth++;
+            numPerLevel[d+1] = numPerLevel[d+1] << 1;
+        if(numPerLevel[0] == 1) {
+            branch = branch >> 1;
+            if (branch<4) branch = 4;
+        }
+    }

I realize that this is a pre-existing issue, but what's so special about 4? Can we add a comment explaining that, and maybe make a constant (max_branch or similar) to hold this special number? Is this just a heuristic value used when we have affinity enabled but don't actually have affinity information?
[Terry L Wilmarth] Yes, this is just a way to put a hierarchy in place when someone changes the default barrier to ‘hierarchical’ on a machine where there is no affinity.  The branching factor of 4 with 4 leaves gives a simple but similar-performing behavior to the hyper barrier’s default settings.  I can add comments and constants along with my other refactoring.
My refactoring consists of moving all the hierarchy stuff outside of the #ifdef for affinity presence, since I need it in general to handle cases with and without affinity.  I think it should look a lot better then, and take fewer LOC.

Thanks again,
From: "Terry L Wilmarth" <terry.l.wilmarth at intel.com<mailto:terry.l.wilmarth at intel.com>>
To: "Terry L Wilmarth" <terry.l.wilmarth at intel.com<mailto:terry.l.wilmarth at intel.com>>, openmp-dev at dcs-maillist2.engr.illinois.edu<mailto:openmp-dev at dcs-maillist2.engr.illinois.edu>
Sent: Friday, April 10, 2015 4:08:01 PM
Subject: Re: [Openmp-dev] Hierarchy generation patch

From: openmp-dev-bounces at cs.uiuc.edu<mailto:openmp-dev-bounces at cs.uiuc.edu> [mailto:openmp-dev-bounces at cs.uiuc.edu] On Behalf Of Wilmarth, Terry L
Sent: Friday, April 03, 2015 11:57 AM
To: openmp-dev at dcs-maillist2.engr.illinois.edu<mailto:openmp-dev at dcs-maillist2.engr.illinois.edu>
Subject: [Openmp-dev] Hierarchy generation patch

Here’s another patch to the hierarchy generation:
1. hierarchy_gen_fixed.patch:  These are improvements to generation of the hierarchy used by hierarchical barrier. They improve how the generation reacts to affinity set to none, or disabled, or no affinity available, or oversubscription.
Terry L. Wilmarth
terry.l.wilmarth at intel.com<mailto:terry.l.wilmarth at intel.com>   217/403-4251
Intel/SSG/DPD/TCAR/RAD/Threading Runtimes

Openmp-dev mailing list
Openmp-dev at dcs-maillist2.engr.illinois.edu<mailto:Openmp-dev at dcs-maillist2.engr.illinois.edu>

Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/openmp-dev/attachments/20150413/c6952fec/attachment.html>

More information about the Openmp-dev mailing list