<html>
<head>
<base href="https://bugs.llvm.org/">
</head>
<body><table border="1" cellspacing="0" cellpadding="8">
<tr>
<th>Bug ID</th>
<td><a class="bz_bug_link
bz_status_NEW "
title="NEW - OMP_PLACES/OMP_PROC_BIND not respecting Sub-NUMA Clustering"
href="https://bugs.llvm.org/show_bug.cgi?id=50681">50681</a>
</td>
</tr>
<tr>
<th>Summary</th>
<td>OMP_PLACES/OMP_PROC_BIND not respecting Sub-NUMA Clustering
</td>
</tr>
<tr>
<th>Product</th>
<td>OpenMP
</td>
</tr>
<tr>
<th>Version</th>
<td>unspecified
</td>
</tr>
<tr>
<th>Hardware</th>
<td>PC
</td>
</tr>
<tr>
<th>OS</th>
<td>Linux
</td>
</tr>
<tr>
<th>Status</th>
<td>NEW
</td>
</tr>
<tr>
<th>Severity</th>
<td>normal
</td>
</tr>
<tr>
<th>Priority</th>
<td>P
</td>
</tr>
<tr>
<th>Component</th>
<td>Runtime Library
</td>
</tr>
<tr>
<th>Assignee</th>
<td>unassignedbugs@nondot.org
</td>
</tr>
<tr>
<th>Reporter</th>
<td>jannis.klinkenberg@rwth-aachen.de
</td>
</tr>
<tr>
<th>CC</th>
<td>llvm-bugs@lists.llvm.org
</td>
</tr></table>
<p>
<div>
<pre>Hi,
I wondering about the semantics for spread binding for OpenMP threads on
machines with miltiple NUMA domains.
We have the following Intel machines with Sub-NUMA Clustering enabled and HT
disabled.
```
$ numactl -H | grep -e cpus -e nodes
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 6 7 8 12 13 14 18 19 20
node 1 cpus: 3 4 5 9 10 11 15 16 17 21 22 23
node 2 cpus: 24 25 26 30 31 32 36 37 38 42 43 44
node 3 cpus: 27 28 29 33 34 35 39 40 41 45 46 47
```
Initially, I assumed an application that is executed with `OMP_NUM_THREADS=4
OMP_PLACES=cores OMP_PROC_BIND=spread` would distribute the OpenMP threads
equally across the NUMA nodes.
This is not the case as OMP_PLACES=cores seems to generate a contiguous list
0-47 and threads are then places to 0,12,24,36
The only way how to currently solve this issue is by specifying/running:
```
export OMP_PLACES=`numactl -H | grep cpus | awk '(NF>3) {for (i = 4; i <= NF;
i++) printf "%d,", $i}' | sed 's/.$//'`
OMP_NUM_THREADS=4 OMP_PROC_BIND=spread ./app
```
Then threads will be assigned to 0,3,24,27 as intented.
Not sure whether this is a bug or just a weakness of the current way how cores
and spread work together.</pre>
</div>
</p>
<hr>
<span>You are receiving this mail because:</span>
<ul>
<li>You are on the CC list for the bug.</li>
</ul>
</body>
</html>