[Openmp-dev] Task-based application aborting in kmp_alloc.cpp
Joseph Schuchart via Openmp-dev
openmp-dev at lists.llvm.org
Sun Jan 24 00:14:49 PST 2021
For the record: With some help from Joachim I found the culprit in a
missing initialization of buckets in __kmp_dephash_extend. This explains
why this issue only occured with large problem sizes that presumably led
to higher numbers of dependencies, which in turn triggered the hash
table extension. A fix has been committed in
https://reviews.llvm.org/D95167 (thanks Andrey).
Cheers
Joseph
On 1/5/21 11:45 AM, Joseph Schuchart via Openmp-dev wrote:
> Small update: I tried to disable the use of fast memory by defining
> USE_FAST_MEMORY to 0 in kmp.h right before the test for whether it is
> defined. My reasoning is that without fast memory it might be easier to
> spot race conditions and/or memory corruption (memory free'd by one
> thread and accessed by another). Unfortunately, I now get a Segfault
> with the following stacktrace obtained from DDT:
>
> #9 main (argc=<optimized out>, argv=<optimized out>) at
> src/npb3.3/BT-MZ-CXX-omptasks/bt.cc:210 (at 0x00000000005067e1)
> #8 __kmpc_fork_call (loc=0x56ba00, argc=15, microtask=0x50c880
> <.omp_outlined.(void) const>) at
> src/clang/llvm-project/openmp/runtime/src/kmp_csupport.cpp:307 (at
> 0x00001555546ae30d)
> #7 __kmp_fork_call (loc=loc at entry=0x56ba00, gtid=gtid at entry=0,
> call_context=call_context at entry=fork_context_intel, argc=argc at entry=15,
> microtask=microtask at entry=0x50c880 <.omp_outlined.(void) const>,
> invoker=<optimized out>, ap=<optimized out>) at
> src/clang/llvm-project/openmp/runtime/src/kmp_runtime.cpp:2240 (at
> 0x00001555546c9ca7)
> #6 __kmp_invoke_task_func (gtid=0) at
> src/clang/llvm-project/openmp/runtime/src/kmp_runtime.cpp:7201 (at
> 0x00001555546c46d8)
> #5 __kmp_invoke_microtask () at
> src/clang/llvm-project/openmp/runtime/src/z_Linux_asm.S:1166 (at
> 0x0000155554746f83)
> #4 .omp_outlined.(void) const (.global_tid.=<optimized out>,
> .bound_tid.=<optimized out>, u=..., qbc_ou=..., qbc_in=..., nx=...,
> nxmax=..., ny=..., nz=..., rho_i=..., us=..., vs=..., ws=..., qs=...,
> square=..., rhs=..., forcing=...) at
> src/npb3.3/BT-MZ-CXX-omptasks/bt.cc:210 (at 0x000000000050c992)
> #3 .omp_outlined._debug__ (.global_tid.=<optimized out>,
> .bound_tid.=<optimized out>, u=..., qbc_ou=..., qbc_in=..., nx=...,
> nxmax=..., ny=..., nz=..., rho_i=..., us=..., vs=..., ws=..., qs=...,
> square=..., rhs=..., forcing=...) at
> src/npb3.3/BT-MZ-CXX-omptasks/bt.cc:214 (at 0x000000000050c992)
> #2 exch_qbc (u=..., qbc_ou=..., qbc_in=..., nx=..., nxmax=..., ny=...,
> nz=..., iprn_msg=<optimized out>) at
> src/npb3.3/BT-MZ-CXX-omptasks/exch_qbc.cc:76 (at 0x00000000005441c8)
> #1 __kmpc_omp_task_with_deps (loc_ref=0x56fa80, gtid=0,
> new_task=0x1551b3708760, ndeps=1, dep_list=0x7fffffff8810,
> ndeps_noalias=0, noalias_dep_list=0x0) at
> src/clang/llvm-project/openmp/runtime/src/kmp_taskdeps.cpp:601 (at
> 0x000015555473ed1f)
> #0 __kmp_init_node (node=0x1551b37065d0) at
> src/clang/llvm-project/openmp/runtime/src/kmp_taskdeps.cpp:601 (at
> 0x000015555473ed1f)
>
> Clang's ASAN reports:
>
> SUMMARY: AddressSanitizer: SEGV
> src/clang/llvm-project/openmp/runtime/src/kmp_taskdeps.cpp:37:23 in
> __kmp_init_node
>
> Interestingly, the address does not seem to be NULL (from looking at it
> in DDT). It seems that __kmp_thread_malloc is not providing valid
> memory? Is setting USE_FAST_MEMORY to 0 in kmp.h the right approach to
> disabling fast memory? I'm not at all familiar with this code base so
> I'm fishing in muddy waters here...
>
> Cheers
> Joseph
>
> On 1/4/21 1:27 PM, Joseph Schuchart via Openmp-dev wrote:
>> Dear devs,
>>
>> Happy new year to all of you!
>>
>> I am seeing the following assert triggering in the Clang OpenMP
>> runtime when running an application using OpenMP tasks and taskloops:
>>
>> Assertion failure at kmp_alloc.cpp(2012): size > 128 * 64.
>> OMP: Error #13: Assertion failure at kmp_alloc.cpp(2012).
>>
>> backtrace from DDT:
>>
>> #19 __kmp_launch_worker (thr=0xa24500) at
>> src/clang/llvm-project/openmp/runtime/src/z_Linux_util.cpp:590 (at
>> 0x0000155554d589ec)
>> #18 __kmp_launch_thread (this_thr=this_thr at entry=0xa24500) at
>> src/clang/llvm-project/openmp/runtime/src/kmp_runtime.cpp:5742 (at
>> 0x0000155554cf03e2)
>> #17 __kmp_fork_barrier (gtid=gtid at entry=26, tid=tid at entry=-2) at
>> src/clang/llvm-project/openmp/runtime/src/kmp_barrier.cpp:1979 (at
>> 0x0000155554d25f0c)
>> #16 __kmp_hyper_barrier_release (bt=bt at entry=bs_forkjoin_barrier,
>> this_thr=this_thr at entry=0xa24500, gtid=gtid at entry=26,
>> tid=tid at entry=-2, propagate_icvs=propagate_icvs at entry=1,
>> itt_sync_obj=itt_sync_obj at entry=0x0) at
>> src/clang/llvm-project/openmp/runtime/src/kmp_barrier.cpp:672 (at
>> 0x0000155554d1c15e)
>> #15 kmp_flag_64<false, true>::wait (itt_sync_obj=0x0, final_spin=1,
>> this_thr=0xa24500, this=0x15514afdba10) at
>> src/clang/llvm-project/openmp/runtime/src/kmp_wait_release.h:899 (at
>> 0x0000155554d1c15e)
>> #14 __kmp_wait_template<kmp_flag_64<>, true, false, true>(kmp_info_t
>> *, kmp_flag_64<false, true> *, void *)
>> (this_thr=this_thr at entry=0xa24500, flag=flag at entry=0x15514afdba10,
>> itt_sync_obj=itt_sync_obj at entry=0x0) at
>> src/clang/llvm-project/openmp/runtime/src/kmp_wait_release.h:345 (at
>> 0x0000155554d1bb29)
>> #13 kmp_flag_64<false, true>::execute_tasks (is_constrained=0,
>> itt_sync_obj=0x0, thread_finished=0x15514afdb94c, final_spin=1,
>> gtid=26, this_thr=0xa24500, this=0x15514afdba10) at
>> src/clang/llvm-project/openmp/runtime/src/kmp_wait_release.h:892 (at
>> 0x0000155554d1bb29)
>> #12 __kmp_execute_tasks_64<false, true> (thread=thread at entry=0xa24500,
>> gtid=gtid at entry=26, flag=flag at entry=0x15514afdba10,
>> final_spin=final_spin at entry=1,
>> thread_finished=thread_finished at entry=0x15514afdb94c,
>> itt_sync_obj=itt_sync_obj at entry=0x0, is_constrained=0) at
>> src/clang/llvm-project/openmp/runtime/src/kmp_tasking.cpp:3029 (at
>> 0x0000155554d176a9)
>> #11 __kmp_execute_tasks_template<kmp_flag_64<> > (is_constrained=0,
>> itt_sync_obj=0x0, thread_finished=<optimized out>,
>> final_spin=<optimized out>, flag=0x15514afdba10, gtid=26,
>> thread=thread at entry=0xa24500) at
>> /sw/hawk-rh8/hlrs/non-spack/compiler/gcc/10.2.0/include/c++/10.2.0/bits/atomic_base.h:420
>> (at 0x0000155554d176a9)
>> #10 __kmp_invoke_task (gtid=gtid at entry=26, task=0xada040,
>> current_task=current_task at entry=0x9b2280) at
>> src/clang/llvm-project/openmp/runtime/src/kmp_tasking.cpp:1589 (at
>> 0x0000155554d10da7)
>> #9 __kmp_task_finish<false> (gtid=gtid at entry=26,
>> task=task at entry=0xada040, resumed_task=resumed_task at entry=0x9b2280) at
>> /sw/hawk-rh8/hlrs/non-spack/compiler/gcc/10.2.0/include/c++/10.2.0/bits/atomic_base.h:556
>> (at 0x0000155554d108e7)
>> #8 __kmp_release_deps (gtid=gtid at entry=26, task=task at entry=0xad9f00)
>> at src/clang/llvm-project/openmp/runtime/src/kmp_taskdeps.h:106 (at
>> 0x0000155554d0fa53)
>> #7 __kmp_dephash_free (h=<optimized out>, thread=0xa24500) at
>> src/clang/llvm-project/openmp/runtime/src/kmp_taskdeps.h:80 (at
>> 0x0000155554d0fa53)
>> #6 __kmp_dephash_free_entries (h=<optimized out>, thread=0xa24500) at
>> src/clang/llvm-project/openmp/runtime/src/kmp_taskdeps.h:62 (at
>> 0x0000155554d0fa53)
>> #5 __kmp_depnode_list_free (list=0x15513affd580, thread=0xa24500) at
>> src/clang/llvm-project/openmp/runtime/src/kmp_taskdeps.h:47 (at
>> 0x0000155554d0fa53)
>> #4 ___kmp_fast_free (this_thr=this_thr at entry=0xa24500,
>> ptr=ptr at entry=0x15513affd580, _file_=_file_ at entry=0x155554d8b710
>> "src/clang/llvm-project/openmp/runtime/src/kmp_taskdeps.h",
>> _line_=_line_ at entry=47) at
>> src/clang/llvm-project/openmp/runtime/src/kmp_alloc.cpp:2012 (at
>> 0x0000155554cc9098)
>> #3 __kmp_debug_assert (msg=msg at entry=0x155554d7e46c "size > 128 * 64",
>> file=0x155554d7d087 "kmp_alloc.cpp", file at entry=0x155554d7d038
>> "src/clang/llvm-project/openmp/runtime/src/kmp_alloc.cpp",
>> line=line at entry=2012) at
>> src/clang/llvm-project/openmp/runtime/src/kmp_debug.cpp:74 (at
>> 0x0000155554ce2b55)
>> #2 __kmp_fatal (message=...) at
>> src/clang/llvm-project/openmp/runtime/src/kmp_i18n.cpp:868 (at
>> 0x0000155554ce77e0)
>> #1 __kmp_abort_process () at
>> src/clang/llvm-project/openmp/runtime/src/kmp_runtime.cpp:444 (at
>> 0x0000155554cf1abc)
>> #0 abort () from /lib64/libc.so.6 (at 0x00001555546e2bce)
>>
>>
>> Any idea what might cause this? I am having a hard time reproducing
>> this in a small example so I'd appreciate if someone could hint at
>> what might trigger this assertion. This error is reproducible in my
>> C++ port of the NPB BT-MZ benchmark using OpenMP tasks (both tied and
>> untied, no detached tasks or offloading), starting from class E (class
>> D runs fine but is too small for my experiments). I have removed all
>> stack-based array allocations to rule out stack overflows inside tasks.
>>
>> This is llvmorg-12-init-15664-gba82c0b315 (pulled last night) compiled
>> with GCC 10.2.0 on an AMD EPYC system running 32 threads per process
>> (16 processes in total using MPI).
>>
>> I can compile and run the application with no issues with GCC 10.2.0.
>> I can reproduce the assert if imposing Clang's libgomp after compiling
>> the application with GCC. I used GCC's ASAN both with GCC's and
>> Clang's libgomp on the application and didn't see any errors. Any
>> advice on what else I could try?
>>
>> Many thanks in advance,
>> Joseph
> _______________________________________________
> Openmp-dev mailing list
> Openmp-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev
More information about the Openmp-dev
mailing list