<div dir="ltr"><div>Adhityaa -</div><div><br></div>Indeed, I would like to see data for interesting tests, not code that doesn't do anything and is pathologically load-imbalanced.<div><br></div><div><a href="https://github.com/ParRes/Kernels">https://github.com/ParRes/Kernels</a> contain some interesting tests that are very easy to run. See p2p*task*, stencil*task*, and transpose*task* in C1z, Cxx11, and/or FORTRAN subdirectories. The language should not matter, as they are designed to be as similar as possible. Configuring the build is just "ln -sf common/make.defs.${toolchain} common/make.defs" after appropriate changes to make.defs.${toolchain}.</div><div><br></div><div>What impact does "very passive" have on these programs? Feel free to contact me privately about experimental setup.<br><br>Jeff<br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 22, 2017 at 9:38 AM, Adhityaa Chandrasekar via Openmp-dev <span dir="ltr"><<a href="mailto:openmp-dev@lists.llvm.org" target="_blank">openmp-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="gmail-">On Fri, Sep 22, 2017 at 9:26 PM, Churbanov, Andrey <span dir="ltr"><<a href="mailto:Andrey.Churbanov@intel.com" target="_blank">Andrey.Churbanov@intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="RU">
<div class="gmail-m_7674011891071793124m_-9000515208982358189WordSection1"><span>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif">> having a thread run at 100% is detrimental to other unrelated processes in the same machine.<u></u><u></u></span></p>
</span><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">You could limit the HW resources for your application (e.g. using KMP_HW_SUBSET environment variable), leaving the rest
of machine for other processes. The OpenMP runtime library has default settings those give as much performance as possible, and these settings are in contrary with composability with other applications running in parallel on the same HW. You may need to manually
adjust settings so that your application behave in composable way.<u></u><u></u></span></p><span>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">></span><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif"> Is this intended? Are there any
other parameters to tweak?<u></u><u></u></span></p>
</span><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Yes, it is intended. If there are no tasks being executed and no tasks to execute, then the blocktime works for idle threads.
If there are tasks then threads never go to sleep waiting for more tasks to be scheduled. That is usual tasking scenario – when one task can generate many more tasks. If we would let idle threads to go to sleep when some thread generates new tasks, that
will slow down a lot of existing codes. So I don’t see how your highly unbalanced case can be improved by the library without hurting real applications with better work balance.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Regarding the wait policy, it is currently implemented so that threads call sched_yield for passive policy and spin more
actively otherwise. Unfortunately modern kernels behave so that yielding threads are scheduled for execution more frequently in order to let all threads consume equal CPU time, regardless of possible work imbalance in the application. We may think of re-implementing
the passive wait policy, or introducing new “very passive” policy, but this would need careful design and time for implementation.</span></p></div></div></blockquote><div><br></div></span><div>Just to clarify: would this new "very passive" (dormant would be a better term IMO) policy be akin to GCC's passive (where they use zero resources because of a futex wait)?</div><span class="gmail-"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="RU"><div class="gmail-m_7674011891071793124m_-9000515208982358189WordSection1"><p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Your current suggested change breaks the task stealing algorithm, because threads do multiple attempts to steal tasks
from randomly chosen victim thread until success, and the change breaks this letting a thread go to sleep after the first stealing attempt. It would be possible to count number of tasks globally and let threads go to sleep if there are no tasks to execute.
But I suspect this can cause significant slowdown of existing codes, so a lot of performance testing should be done first before applying something like this.</span></p></div></div></blockquote><div><br></div></span><div>Does it have to be global? Making it global would mean a lot of cache misses, right? Have a per-thread counter that's decremented every time it can't steal a task, if it reaches zero reset to default value and go to sleep. At any point in time, if we steal a task successfully, reset the counter.</div><div><div class="gmail-h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="RU"><div class="gmail-m_7674011891071793124m_-9000515208982358189WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-- Andrey<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif"> Openmp-dev [mailto:<a href="mailto:openmp-dev-bounces@lists.llvm.org" target="_blank">openmp-dev-bounces@lis<wbr>ts.llvm.org</a>]
<b>On Behalf Of </b>Adhityaa Chandrasekar via Openmp-dev<br>
<b>Sent:</b> Friday, September 22, 2017 9:22 AM<span><br>
<b>To:</b> Jeff Hammond <<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a>><br>
<b>Cc:</b> <a href="mailto:openmp-dev@lists.llvm.org" target="_blank">openmp-dev@lists.llvm.org</a><br>
</span><span><b>Subject:</b> Re: [Openmp-dev] 100% CPU usage when threads can't steal tasks<u></u><u></u></span></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Fri, Sep 22, 2017 at 2:21 AM, Jeff Hammond <<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a>> wrote:<u></u><u></u></p><div><div class="gmail-m_7674011891071793124h5">
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">You might find <a href="https://software.intel.com/en-us/forums/intel-many-integrated-core/topic/556869" target="_blank">https://software.intel.co<wbr>m/en-us/forums/intel-many-inte<wbr>grated-core/topic/556869</a> relevant.<u></u><u></u></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks for that link. I agree with the statement that time to solution is a more important metric in comparison to CPU time. However, having a thread run at 100% is detrimental to other unrelated processes in the same machine.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Is there anything else you want me to notice in that thread?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I consider idling in the OpenMP runtime to be a sign of a badly behaved application, not a runtime bug in need of fixing. But in any case, OMP_WAIT_POLICY/KMP_BLOCKTIME exist to address that.<u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">OMP_WAIT_POLICY and KMP_BLOCKTIME indeed are designed to solve that - however, I don't think the clang OpenMP runtime is respecting the OMP_WAIT_POLICY value -- I set `export OMP_WAIT_POLICY=passive` and compiled the example program with
g++ and clang++. The g++ binary used virtually no CPU while the clang++ binary had three processes at 100% (three because OMP_NUM_THREADS=4 and only 1 is executing a task; the other three are idle).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Is this intended? Are there any other parameters to tweak?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">In any case, I think a default of "temporarily active for a while before switching to passive" instead of "always active unless the user manually overrides" is saner, no?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Best,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Jeff<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">On Thu, Sep 21, 2017 at 1:11 PM, Adhityaa Chandrasekar via Openmp-dev <<a href="mailto:openmp-dev@lists.llvm.org" target="_blank">openmp-dev@lists.llvm.org</a>> wrote:<u></u><u></u></p>
</div>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">Hi,</span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">My name is Adhityaa and I'm an undergrad student. This is my first time getting involved in OpenMP development at LLVM. I have used the library a bunch of times before, but I haven't been involved in the behind-the-scenes
work before.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">So I was just taking a look at <a href="https://bugs.llvm.org/show_bug.cgi?id=33543" target="_blank">https://bugs.llvm.org/show_<wbr>bug.cgi?id=33543</a> and I tried solving it (it looked like a simple, reproducible
bug that looked quite important). First I ran strace on the program to see exactly what was happening - I found that the threads without any tasks were issuing a whole bunch of `sched_yield()`, making the CPU go to 100%.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">Then I tried going into the code. I came up with a fairly simple patch that almost solved the issue:<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt"><u></u> <u></u></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New"">--- kmp_wait_release.h (revision 313888)</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New"">+++ kmp_wait_release.h (working copy)</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New"">@@ -188,6 +188,7 @@</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> // Main wait spin loop</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> while (flag->notdone_check()) {</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> int in_pool;</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New"">+ int executed_tasks = 1;</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> kmp_task_team_t *task_team = NULL;</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> if (__kmp_tasking_mode != tskm_immediate_exec) {</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> task_team = this_thr->th.th_task_team;</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New"">@@ -200,10 +201,11 @@</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> disabled (KMP_TASKING=0). */</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> if (task_team != NULL) {</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> if (TCR_SYNC_4(task_team->tt.tt_a<wbr>ctive)) {</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New"">- if (KMP_TASKING_ENABLED(task_team<wbr>))</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New"">- flag->execute_tasks(</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New"">+ if (KMP_TASKING_ENABLED(task_team<wbr>)) {</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New"">+ executed_tasks = flag->execute_tasks(</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> this_thr, th_gtid, final_spin,</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> &tasks_completed USE_ITT_BUILD_ARG(itt_sync_obj<wbr>), 0);</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New"">+ }</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> else</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> this_thr->th.th_reap_state = KMP_SAFE_TO_REAP;</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> } else {</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New"">@@ -269,7 +271,7 @@</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> continue;</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> </span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> // Don't suspend if there is a likelihood of new tasks being spawned.</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New"">- if ((task_team != NULL) && TCR_4(task_team->tt.tt_found_t<wbr>asks))</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New"">+ if ((task_team != NULL) && TCR_4(task_team->tt.tt_found_t<wbr>asks) && executed_tasks)</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> continue;</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> </span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New""> #if KMP_USE_MONITOR</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">Alas, this led to a deadlock - the thread went into a futex wait and was never woke again. So I looked at how GCC did things and they issue a FUTEX_WAKE_PRIVATE for INT_MAX threads at the end of things. This
should solve the problem, I think?<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">Anyway, it's pretty late here and I've been at this for over 6-7 hours at a stretch, so I'm really tired. I was just wondering if I could get some help on how to proceed from here (and whether I'm even on the
right track).<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">Thanks,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">Adhityaa<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12pt">______________________________<wbr>_________________<br>
Openmp-dev mailing list<br>
<a href="mailto:Openmp-dev@lists.llvm.org" target="_blank">Openmp-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/openmp-dev</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><span style="color:rgb(136,136,136)"><br>
<br clear="all">
<span class="gmail-m_7674011891071793124m_-9000515208982358189hoenzb"><u></u><u></u></span></span></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><span class="gmail-m_7674011891071793124m_-9000515208982358189hoenzb"><span style="color:rgb(136,136,136)">-- </span><u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="color:rgb(136,136,136)">Jeff Hammond<br>
<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br>
<a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></span><u></u><u></u></p>
</div>
</div>
</blockquote>
</div></div></div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
<p><br>------------------------------<wbr>------------------------------<wbr>--------<br>Joint Stock Company Intel A/O<br>Registered legal address: Krylatsky Hills Business Park, <br><a href="https://maps.google.com/?q=17+Krylatskaya+Str.,+Bldg+4,+Moscow+121614,+Russian+Federation&entry=gmail&source=g" target="_blank">17 Krylatskaya Str., Bldg 4, Moscow 121614, </a><br><a href="https://maps.google.com/?q=17+Krylatskaya+Str.,+Bldg+4,+Moscow+121614,+Russian+Federation&entry=gmail&source=g" target="_blank">Russian Federation</a></p><span><p>This e-mail and any attachments may contain confidential material for<br>the sole use of the intended recipient(s). Any review or distribution<br>by others is strictly prohibited. If you are not the intended<br>recipient, please contact the sender and delete all copies.</p>
</span></div>
</blockquote></div></div></div><br></div></div>
<br>______________________________<wbr>_________________<br>
Openmp-dev mailing list<br>
<a href="mailto:Openmp-dev@lists.llvm.org">Openmp-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/openmp-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</div></div></div>