[llvm-bugs] [Bug 46832] New: qsort() interceptor broken on FreeBSD

[via llvm-bugs]

https://bugs.llvm.org/show_bug.cgi?id=46832

            Bug ID: 46832
           Summary: qsort() interceptor broken on FreeBSD
           Product: compiler-rt
           Version: unspecified
          Hardware: PC
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P
         Component: asan
          Assignee: unassignedbugs at nondot.org
          Reporter: arichardson.kde at gmail.com
                CC: llvm-bugs at lists.llvm.org

The qsort interceptor results in an infinite loop on FreeBSD as can be seen
from this backtrace:

0x00000000016b6d06 in wrapped_qsort_compar(void const*, void const*) ()
    at
/usr/local/poudriere/ports/brooks/devel/llvm-devel/work/llvm-project-61c2a0bb823677ce0e604b92e5dae65d9bd32b6e/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:9690
9690      return qsort_compar(a, b);
(gdb) bt
#0  0x00000000016b6d06 in wrapped_qsort_compar(void const*, void const*) ()
    at
/usr/local/poudriere/ports/brooks/devel/llvm-devel/work/llvm-project-61c2a0bb823677ce0e604b92e5dae65d9bd32b6e/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:9690
#1  0x0000000802efaa70 in qsort () from /lib/libc.so.7
#2  0x00000000016b6df3 in __interceptor_qsort ()
    at
/usr/local/poudriere/ports/brooks/devel/llvm-devel/work/llvm-project-61c2a0bb823677ce0e604b92e5dae65d9bd32b6e/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:9710
#3  0x0000000802efa99f in qsort () from /lib/libc.so.7
#4  0x00000000016b6df3 in __interceptor_qsort ()
    at
/usr/local/poudriere/ports/brooks/devel/llvm-devel/work/llvm-project-61c2a0bb823677ce0e604b92e5dae65d9bd32b6e/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:9710
#5  0x0000000001e9284f in llvm::array_pod_sort<std::__1::__wrap_iter<(anonymous
namespace)::ELFWriter::ELFSymbolData*> > (Start=..., End=...)
    at
/exports/users/alr48/sources/llvm-project/llvm/include/llvm/ADT/STLExtras.h:1399
#6  0x0000000001e81e83 in (anonymous namespace)::ELFWriter::computeSymbolTable
(this=0x7fffffffbf20, Asm=...,
    Layout=..., SectionIndexMap=..., RevGroupMap=..., SectionOffsets=...)


The problem is that qsort() calls qsort() on individual parts of the array
which ends up calling the interceptor again. However, the interceptor uses a
thread local function pointer for the comparator wrapper.

The interceptor code does not handle this case:

  qsort_compar_f old_compar = qsort_compar;
  qsort_compar = compar;
  SIZE_T old_size = qsort_size;
  qsort_size = size;
  REAL(qsort)(base, nmemb, size, wrapped_qsort_compar);
  qsort_compar = old_compar;
  qsort_size = old_size;

In the first call it will assign the real comparator to qsort_compar, but in
the recursive call from inside qsort qsort_compar will end up being set to
wrapped_qsort_compar and it ends up being an infinite loop.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-bugs/attachments/20200724/a458372f/attachment.html>