[Openmp-dev] 100% CPU usage when threads can't steal tasks
Adhityaa Chandrasekar via Openmp-dev
openmp-dev at lists.llvm.org
Fri Sep 22 09:38:37 PDT 2017
On Fri, Sep 22, 2017 at 9:26 PM, Churbanov, Andrey <
Andrey.Churbanov at intel.com> wrote:
> > having a thread run at 100% is detrimental to other unrelated processes
> in the same machine.
>
> You could limit the HW resources for your application (e.g. using
> KMP_HW_SUBSET environment variable), leaving the rest of machine for other
> processes. The OpenMP runtime library has default settings those give as
> much performance as possible, and these settings are in contrary with
> composability with other applications running in parallel on the same HW.
> You may need to manually adjust settings so that your application behave in
> composable way.
>
>
>
> > Is this intended? Are there any other parameters to tweak?
>
> Yes, it is intended. If there are no tasks being executed and no tasks to
> execute, then the blocktime works for idle threads. If there are tasks
> then threads never go to sleep waiting for more tasks to be scheduled.
> That is usual tasking scenario – when one task can generate many more
> tasks. If we would let idle threads to go to sleep when some thread
> generates new tasks, that will slow down a lot of existing codes. So I
> don’t see how your highly unbalanced case can be improved by the library
> without hurting real applications with better work balance.
>
>
>
> Regarding the wait policy, it is currently implemented so that threads
> call sched_yield for passive policy and spin more actively otherwise.
> Unfortunately modern kernels behave so that yielding threads are scheduled
> for execution more frequently in order to let all threads consume equal CPU
> time, regardless of possible work imbalance in the application. We may
> think of re-implementing the passive wait policy, or introducing new “very
> passive” policy, but this would need careful design and time for
> implementation.
>
Just to clarify: would this new "very passive" (dormant would be a better
term IMO) policy be akin to GCC's passive (where they use zero resources
because of a futex wait)?
>
>
> Your current suggested change breaks the task stealing algorithm, because
> threads do multiple attempts to steal tasks from randomly chosen victim
> thread until success, and the change breaks this letting a thread go to
> sleep after the first stealing attempt. It would be possible to count
> number of tasks globally and let threads go to sleep if there are no tasks
> to execute. But I suspect this can cause significant slowdown of existing
> codes, so a lot of performance testing should be done first before applying
> something like this.
>
Does it have to be global? Making it global would mean a lot of cache
misses, right? Have a per-thread counter that's decremented every time it
can't steal a task, if it reaches zero reset to default value and go to
sleep. At any point in time, if we steal a task successfully, reset the
counter.
>
>
> -- Andrey
>
>
>
> *From:* Openmp-dev [mailto:openmp-dev-bounces at lists.llvm.org] *On Behalf
> Of *Adhityaa Chandrasekar via Openmp-dev
> *Sent:* Friday, September 22, 2017 9:22 AM
> *To:* Jeff Hammond <jeff.science at gmail.com>
> *Cc:* openmp-dev at lists.llvm.org
> *Subject:* Re: [Openmp-dev] 100% CPU usage when threads can't steal tasks
>
>
>
>
>
> On Fri, Sep 22, 2017 at 2:21 AM, Jeff Hammond <jeff.science at gmail.com>
> wrote:
>
> You might find https://software.intel.com/en-us/forums/intel-many-
> integrated-core/topic/556869 relevant.
>
>
>
> Thanks for that link. I agree with the statement that time to solution is
> a more important metric in comparison to CPU time. However, having a thread
> run at 100% is detrimental to other unrelated processes in the same machine.
>
>
>
> Is there anything else you want me to notice in that thread?
>
>
>
>
>
> I consider idling in the OpenMP runtime to be a sign of a badly behaved
> application, not a runtime bug in need of fixing. But in any case,
> OMP_WAIT_POLICY/KMP_BLOCKTIME exist to address that.
>
>
>
> OMP_WAIT_POLICY and KMP_BLOCKTIME indeed are designed to solve that -
> however, I don't think the clang OpenMP runtime is respecting the
> OMP_WAIT_POLICY value -- I set `export OMP_WAIT_POLICY=passive` and
> compiled the example program with g++ and clang++. The g++ binary used
> virtually no CPU while the clang++ binary had three processes at 100%
> (three because OMP_NUM_THREADS=4 and only 1 is executing a task; the other
> three are idle).
>
>
>
> Is this intended? Are there any other parameters to tweak?
>
>
>
> In any case, I think a default of "temporarily active for a while before
> switching to passive" instead of "always active unless the user manually
> overrides" is saner, no?
>
>
>
>
>
> Best,
>
>
>
> Jeff
>
>
>
> On Thu, Sep 21, 2017 at 1:11 PM, Adhityaa Chandrasekar via Openmp-dev <
> openmp-dev at lists.llvm.org> wrote:
>
> Hi,
>
>
>
> My name is Adhityaa and I'm an undergrad student. This is my first time
> getting involved in OpenMP development at LLVM. I have used the library a
> bunch of times before, but I haven't been involved in the behind-the-scenes
> work before.
>
>
>
> So I was just taking a look at https://bugs.llvm.org/show_bug.cgi?id=33543 and
> I tried solving it (it looked like a simple, reproducible bug that looked
> quite important). First I ran strace on the program to see exactly what was
> happening - I found that the threads without any tasks were issuing a whole
> bunch of `sched_yield()`, making the CPU go to 100%.
>
>
>
> Then I tried going into the code. I came up with a fairly simple patch
> that almost solved the issue:
>
>
>
> --- kmp_wait_release.h (revision 313888)
>
> +++ kmp_wait_release.h (working copy)
>
> @@ -188,6 +188,7 @@
>
> // Main wait spin loop
>
> while (flag->notdone_check()) {
>
> int in_pool;
>
> + int executed_tasks = 1;
>
> kmp_task_team_t *task_team = NULL;
>
> if (__kmp_tasking_mode != tskm_immediate_exec) {
>
> task_team = this_thr->th.th_task_team;
>
> @@ -200,10 +201,11 @@
>
> disabled (KMP_TASKING=0). */
>
> if (task_team != NULL) {
>
> if (TCR_SYNC_4(task_team->tt.tt_active)) {
>
> - if (KMP_TASKING_ENABLED(task_team))
>
> - flag->execute_tasks(
>
> + if (KMP_TASKING_ENABLED(task_team)) {
>
> + executed_tasks = flag->execute_tasks(
>
> this_thr, th_gtid, final_spin,
>
> &tasks_completed USE_ITT_BUILD_ARG(itt_sync_obj), 0);
>
> + }
>
> else
>
> this_thr->th.th_reap_state = KMP_SAFE_TO_REAP;
>
> } else {
>
> @@ -269,7 +271,7 @@
>
> continue;
>
>
>
> // Don't suspend if there is a likelihood of new tasks being spawned.
>
> - if ((task_team != NULL) && TCR_4(task_team->tt.tt_found_tasks))
>
> + if ((task_team != NULL) && TCR_4(task_team->tt.tt_found_tasks) &&
> executed_tasks)
>
> continue;
>
>
>
> #if KMP_USE_MONITOR
>
>
>
> Alas, this led to a deadlock - the thread went into a futex wait and was
> never woke again. So I looked at how GCC did things and they issue a
> FUTEX_WAKE_PRIVATE for INT_MAX threads at the end of things. This should
> solve the problem, I think?
>
>
>
> Anyway, it's pretty late here and I've been at this for over 6-7 hours at
> a stretch, so I'm really tired. I was just wondering if I could get some
> help on how to proceed from here (and whether I'm even on the right track).
>
>
>
> Thanks,
>
> Adhityaa
>
>
>
> _______________________________________________
> Openmp-dev mailing list
> Openmp-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev
>
>
>
>
>
> --
>
> Jeff Hammond
> jeff.science at gmail.com
> http://jeffhammond.github.io/
>
>
>
>
> --------------------------------------------------------------------
> Joint Stock Company Intel A/O
> Registered legal address: Krylatsky Hills Business Park,
> 17 Krylatskaya Str., Bldg 4, Moscow 121614,
> <https://maps.google.com/?q=17+Krylatskaya+Str.,+Bldg+4,+Moscow+121614,+Russian+Federation&entry=gmail&source=g>
> Russian Federation
> <https://maps.google.com/?q=17+Krylatskaya+Str.,+Bldg+4,+Moscow+121614,+Russian+Federation&entry=gmail&source=g>
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/openmp-dev/attachments/20170922/db17257e/attachment-0001.html>
More information about the Openmp-dev
mailing list