[Openmp-dev] Task-based application aborting in kmp_alloc.cpp
Joseph Schuchart via Openmp-dev
openmp-dev at lists.llvm.org
Tue Jan 5 02:45:44 PST 2021
Small update: I tried to disable the use of fast memory by defining
USE_FAST_MEMORY to 0 in kmp.h right before the test for whether it is
defined. My reasoning is that without fast memory it might be easier to
spot race conditions and/or memory corruption (memory free'd by one
thread and accessed by another). Unfortunately, I now get a Segfault
with the following stacktrace obtained from DDT:
#9 main (argc=<optimized out>, argv=<optimized out>) at
src/npb3.3/BT-MZ-CXX-omptasks/bt.cc:210 (at 0x00000000005067e1)
#8 __kmpc_fork_call (loc=0x56ba00, argc=15, microtask=0x50c880
<.omp_outlined.(void) const>) at
src/clang/llvm-project/openmp/runtime/src/kmp_csupport.cpp:307 (at
0x00001555546ae30d)
#7 __kmp_fork_call (loc=loc at entry=0x56ba00, gtid=gtid at entry=0,
call_context=call_context at entry=fork_context_intel, argc=argc at entry=15,
microtask=microtask at entry=0x50c880 <.omp_outlined.(void) const>,
invoker=<optimized out>, ap=<optimized out>) at
src/clang/llvm-project/openmp/runtime/src/kmp_runtime.cpp:2240 (at
0x00001555546c9ca7)
#6 __kmp_invoke_task_func (gtid=0) at
src/clang/llvm-project/openmp/runtime/src/kmp_runtime.cpp:7201 (at
0x00001555546c46d8)
#5 __kmp_invoke_microtask () at
src/clang/llvm-project/openmp/runtime/src/z_Linux_asm.S:1166 (at
0x0000155554746f83)
#4 .omp_outlined.(void) const (.global_tid.=<optimized out>,
.bound_tid.=<optimized out>, u=..., qbc_ou=..., qbc_in=..., nx=...,
nxmax=..., ny=..., nz=..., rho_i=..., us=..., vs=..., ws=..., qs=...,
square=..., rhs=..., forcing=...) at
src/npb3.3/BT-MZ-CXX-omptasks/bt.cc:210 (at 0x000000000050c992)
#3 .omp_outlined._debug__ (.global_tid.=<optimized out>,
.bound_tid.=<optimized out>, u=..., qbc_ou=..., qbc_in=..., nx=...,
nxmax=..., ny=..., nz=..., rho_i=..., us=..., vs=..., ws=..., qs=...,
square=..., rhs=..., forcing=...) at
src/npb3.3/BT-MZ-CXX-omptasks/bt.cc:214 (at 0x000000000050c992)
#2 exch_qbc (u=..., qbc_ou=..., qbc_in=..., nx=..., nxmax=..., ny=...,
nz=..., iprn_msg=<optimized out>) at
src/npb3.3/BT-MZ-CXX-omptasks/exch_qbc.cc:76 (at 0x00000000005441c8)
#1 __kmpc_omp_task_with_deps (loc_ref=0x56fa80, gtid=0,
new_task=0x1551b3708760, ndeps=1, dep_list=0x7fffffff8810,
ndeps_noalias=0, noalias_dep_list=0x0) at
src/clang/llvm-project/openmp/runtime/src/kmp_taskdeps.cpp:601 (at
0x000015555473ed1f)
#0 __kmp_init_node (node=0x1551b37065d0) at
src/clang/llvm-project/openmp/runtime/src/kmp_taskdeps.cpp:601 (at
0x000015555473ed1f)
Clang's ASAN reports:
SUMMARY: AddressSanitizer: SEGV
src/clang/llvm-project/openmp/runtime/src/kmp_taskdeps.cpp:37:23 in
__kmp_init_node
Interestingly, the address does not seem to be NULL (from looking at it
in DDT). It seems that __kmp_thread_malloc is not providing valid
memory? Is setting USE_FAST_MEMORY to 0 in kmp.h the right approach to
disabling fast memory? I'm not at all familiar with this code base so
I'm fishing in muddy waters here...
Cheers
Joseph
On 1/4/21 1:27 PM, Joseph Schuchart via Openmp-dev wrote:
> Dear devs,
>
> Happy new year to all of you!
>
> I am seeing the following assert triggering in the Clang OpenMP runtime
> when running an application using OpenMP tasks and taskloops:
>
> Assertion failure at kmp_alloc.cpp(2012): size > 128 * 64.
> OMP: Error #13: Assertion failure at kmp_alloc.cpp(2012).
>
> backtrace from DDT:
>
> #19 __kmp_launch_worker (thr=0xa24500) at
> src/clang/llvm-project/openmp/runtime/src/z_Linux_util.cpp:590 (at
> 0x0000155554d589ec)
> #18 __kmp_launch_thread (this_thr=this_thr at entry=0xa24500) at
> src/clang/llvm-project/openmp/runtime/src/kmp_runtime.cpp:5742 (at
> 0x0000155554cf03e2)
> #17 __kmp_fork_barrier (gtid=gtid at entry=26, tid=tid at entry=-2) at
> src/clang/llvm-project/openmp/runtime/src/kmp_barrier.cpp:1979 (at
> 0x0000155554d25f0c)
> #16 __kmp_hyper_barrier_release (bt=bt at entry=bs_forkjoin_barrier,
> this_thr=this_thr at entry=0xa24500, gtid=gtid at entry=26, tid=tid at entry=-2,
> propagate_icvs=propagate_icvs at entry=1,
> itt_sync_obj=itt_sync_obj at entry=0x0) at
> src/clang/llvm-project/openmp/runtime/src/kmp_barrier.cpp:672 (at
> 0x0000155554d1c15e)
> #15 kmp_flag_64<false, true>::wait (itt_sync_obj=0x0, final_spin=1,
> this_thr=0xa24500, this=0x15514afdba10) at
> src/clang/llvm-project/openmp/runtime/src/kmp_wait_release.h:899 (at
> 0x0000155554d1c15e)
> #14 __kmp_wait_template<kmp_flag_64<>, true, false, true>(kmp_info_t *,
> kmp_flag_64<false, true> *, void *) (this_thr=this_thr at entry=0xa24500,
> flag=flag at entry=0x15514afdba10, itt_sync_obj=itt_sync_obj at entry=0x0) at
> src/clang/llvm-project/openmp/runtime/src/kmp_wait_release.h:345 (at
> 0x0000155554d1bb29)
> #13 kmp_flag_64<false, true>::execute_tasks (is_constrained=0,
> itt_sync_obj=0x0, thread_finished=0x15514afdb94c, final_spin=1, gtid=26,
> this_thr=0xa24500, this=0x15514afdba10) at
> src/clang/llvm-project/openmp/runtime/src/kmp_wait_release.h:892 (at
> 0x0000155554d1bb29)
> #12 __kmp_execute_tasks_64<false, true> (thread=thread at entry=0xa24500,
> gtid=gtid at entry=26, flag=flag at entry=0x15514afdba10,
> final_spin=final_spin at entry=1,
> thread_finished=thread_finished at entry=0x15514afdb94c,
> itt_sync_obj=itt_sync_obj at entry=0x0, is_constrained=0) at
> src/clang/llvm-project/openmp/runtime/src/kmp_tasking.cpp:3029 (at
> 0x0000155554d176a9)
> #11 __kmp_execute_tasks_template<kmp_flag_64<> > (is_constrained=0,
> itt_sync_obj=0x0, thread_finished=<optimized out>, final_spin=<optimized
> out>, flag=0x15514afdba10, gtid=26, thread=thread at entry=0xa24500) at
> /sw/hawk-rh8/hlrs/non-spack/compiler/gcc/10.2.0/include/c++/10.2.0/bits/atomic_base.h:420
> (at 0x0000155554d176a9)
> #10 __kmp_invoke_task (gtid=gtid at entry=26, task=0xada040,
> current_task=current_task at entry=0x9b2280) at
> src/clang/llvm-project/openmp/runtime/src/kmp_tasking.cpp:1589 (at
> 0x0000155554d10da7)
> #9 __kmp_task_finish<false> (gtid=gtid at entry=26,
> task=task at entry=0xada040, resumed_task=resumed_task at entry=0x9b2280) at
> /sw/hawk-rh8/hlrs/non-spack/compiler/gcc/10.2.0/include/c++/10.2.0/bits/atomic_base.h:556
> (at 0x0000155554d108e7)
> #8 __kmp_release_deps (gtid=gtid at entry=26, task=task at entry=0xad9f00) at
> src/clang/llvm-project/openmp/runtime/src/kmp_taskdeps.h:106 (at
> 0x0000155554d0fa53)
> #7 __kmp_dephash_free (h=<optimized out>, thread=0xa24500) at
> src/clang/llvm-project/openmp/runtime/src/kmp_taskdeps.h:80 (at
> 0x0000155554d0fa53)
> #6 __kmp_dephash_free_entries (h=<optimized out>, thread=0xa24500) at
> src/clang/llvm-project/openmp/runtime/src/kmp_taskdeps.h:62 (at
> 0x0000155554d0fa53)
> #5 __kmp_depnode_list_free (list=0x15513affd580, thread=0xa24500) at
> src/clang/llvm-project/openmp/runtime/src/kmp_taskdeps.h:47 (at
> 0x0000155554d0fa53)
> #4 ___kmp_fast_free (this_thr=this_thr at entry=0xa24500,
> ptr=ptr at entry=0x15513affd580, _file_=_file_ at entry=0x155554d8b710
> "src/clang/llvm-project/openmp/runtime/src/kmp_taskdeps.h",
> _line_=_line_ at entry=47) at
> src/clang/llvm-project/openmp/runtime/src/kmp_alloc.cpp:2012 (at
> 0x0000155554cc9098)
> #3 __kmp_debug_assert (msg=msg at entry=0x155554d7e46c "size > 128 * 64",
> file=0x155554d7d087 "kmp_alloc.cpp", file at entry=0x155554d7d038
> "src/clang/llvm-project/openmp/runtime/src/kmp_alloc.cpp",
> line=line at entry=2012) at
> src/clang/llvm-project/openmp/runtime/src/kmp_debug.cpp:74 (at
> 0x0000155554ce2b55)
> #2 __kmp_fatal (message=...) at
> src/clang/llvm-project/openmp/runtime/src/kmp_i18n.cpp:868 (at
> 0x0000155554ce77e0)
> #1 __kmp_abort_process () at
> src/clang/llvm-project/openmp/runtime/src/kmp_runtime.cpp:444 (at
> 0x0000155554cf1abc)
> #0 abort () from /lib64/libc.so.6 (at 0x00001555546e2bce)
>
>
> Any idea what might cause this? I am having a hard time reproducing this
> in a small example so I'd appreciate if someone could hint at what might
> trigger this assertion. This error is reproducible in my C++ port of the
> NPB BT-MZ benchmark using OpenMP tasks (both tied and untied, no
> detached tasks or offloading), starting from class E (class D runs fine
> but is too small for my experiments). I have removed all stack-based
> array allocations to rule out stack overflows inside tasks.
>
> This is llvmorg-12-init-15664-gba82c0b315 (pulled last night) compiled
> with GCC 10.2.0 on an AMD EPYC system running 32 threads per process (16
> processes in total using MPI).
>
> I can compile and run the application with no issues with GCC 10.2.0. I
> can reproduce the assert if imposing Clang's libgomp after compiling the
> application with GCC. I used GCC's ASAN both with GCC's and Clang's
> libgomp on the application and didn't see any errors. Any advice on what
> else I could try?
>
> Many thanks in advance,
> Joseph
More information about the Openmp-dev
mailing list