[PATCH] [TSan][MIPS] Adding support for MIPS64

[Sagar Thakur]

================
Comment at: lib/tsan/rtl/tsan_interceptors.cc:661
@@ +660,3 @@
+#ifdef __mips__
+  // For mips memset gets called before a call to __cxa_guard_aquire
+  // which causes a crash in REAL(pthread_create) during initialization
----------------
samsonov wrote:
> sagar wrote:
> > samsonov wrote:
> > > sagar wrote:
> > > > samsonov wrote:
> > > > > sagar wrote:
> > > > > > samsonov wrote:
> > > > > > > sagar wrote:
> > > > > > > > samsonov wrote:
> > > > > > > > > I don't understand this comment. Why does REAL(pthread_create) crash?
> > > > > > > > As per my observation, in x86_64 first __cxa_guard_aquired() is called and its interceptor calls Initialize().
> > > > > > > > Initialize() creates background thread.
> > > > > > > > But in mips64 __interceptor_memset gets called before __cxa_guard_aquire() interceptor. So Initialize() is called from within __intercerptor_memset and REAL(pthread_create) emits a SIGIOT.
> > > > > > > > 
> > > > > > > > @dvyukov I don't know why such behaviour in mips, can you help us with this?
> > > > > > > Why does the call to REAL(pthread_create) succeeds if it is issued from `__cxa_guard_acquired` interceptor and fails if it is issued from `memset` interceptor? I think you'd have to figure it out.
> > > > > > > 
> > > > > > > I don't see the immediate problem from your description: memset interceptor calls Initialize(), which calls InitializeIntercepotrs() (that is supposed to initialize all REAL(foo) function pointers), and only then calls internal_start_thread.
> > > > > > While debugging the code, I observed that __interceptor_memset and __cxa_guard_acquire are being called even before __start. So its reference is probably in LLVM/clang. I have no knowledge from where they are being called.
> > > > > > 
> > > > > > Through strace in x86_64, the sequence of calls observed:
> > > > > >   rt_sigaction() -> rt_sigprocmask() -> __cxa_guard_acquired() -> Initialize()
> > > > > > 
> > > > > > In case of mips64, the sequence is:
> > > > > >   __interceptor_memset() -> Initialize()
> > > > > > 
> > > > > > So for mips64 there is no signal handler registered during Initialize() because rt_sigaction syscall is not yet issued.
> > > > > > 
> > > > > > I will be very thankful if someone could advice me on how to solve this issue.
> > > > > It's hard to predict which TSan interceptor will be called first (and thus will be first to call __tsan::Initialize()). For example, on my x86_64 host the stack trace is:
> > > > >   __tsan::Initialize
> > > > >   ScopedInterceptor::ScopedInterceptor
> > > > >   __cxa_guard_acquire
> > > > >   __future_category_instance
> > > > >   __static_initialization_and_destruction_0
> > > > >   _GLOBAL__sub_I_compatibility_thread_c__0x.cc
> > > > >   call_init
> > > > >   _dl_start_user
> > > > > 
> > > > > I don't see why signal handler is at all important. Can you just debug your TSan-instrumented program in gdb, reach the place where it fails with SIGIOT/SIGABRT and learn the reason why it's happening? E.g. it calls REAL(foo), but this function pointer is not yet initialized. Or it accesses mprotect()-ed memory. Or something else.
> > > > > 
> > > > > Sorry, I can't yet diagnose the issue from your descriptions.
> > > > Hi,
> > > > 
> > > > The intercepted memset is called during pthread initialization function __pthread_initialize_minimal_internal ().
> > > > 
> > > > Sequence of calls for MIPS:
> > > > 
> > > > ```
> > > >  __pthread_initialize_minimal_internal () -> __sigemptyset () -> __interceptor_memset () -> pthread_create () -> __GI_abort ()
> > > > ```
> > > > 
> > > > The intercepted memset calls pthread_create() before the minimal initialization required for pthread_create is complete and this results in abort.
> > > > On x86, __pthread_initialize_minimal_internal () does not call memset and thus can complete the minimal initialization.
> > > > For MIPS, we need to avoid calling pthread_create () if memset is called during pthread initialization.
> > > So, this has nothing to do with `__cxa_guard_acquired` or `memset`. You are just not allowed to call pthread_create() from TSan runtime before `__pthread_initialize_minimal_internal` exits. For example, you can try to intercept `__pthread_initialize_minimal_internal` and use `ScopedIgnoreInterceptors` to disable TSan interceptors for sigemptyset(), memset() or whatever.
> > Hi,
> > 
> > I tried to intercept `__pthread_initialize_minimal_internal ()`, but it does not get intercepted. As far as I know, this is probably because references to `__pthread_initialize_minimal_internal ()` are resolved when glibc is compiled. I also tried to search for calls to functions occurring before call to `__pthread_initialize_minimal_internal ()` which would be resolved at runtime, so that we could intercept them and use `ScopedIgnoreInterceptors`, but I couldn’t see any such function call.
> Who calls `__pthread_initialize_minimal_internal` at all in that case? Is it called from library constructor? As a quick workaround, you can disable background thread in MIPS port - IIRC it's not critical to TSan functioning.
> Who calls __pthread_initialize_minimal_internal at all in that case?

Here is the backtrace at __pthread_initialize_minimal_internal () :


```
(gdb) bt
#0  __pthread_initialize_minimal_internal () at nptl-init.c:281
#1  0x000000ffed9567e4 in _init () at ../ports/sysdeps/mips/mips64/n64/crti.S:80
#2  0x000000ffed9a0d94 in call_init (l=0xffed9c0930, argc=argc at entry=1, argv=argv at entry=0xffff77a658, env=env at entry=0xffff77a668)
    at dl-init.c:64
#3  0x000000ffed9a0fc0 in call_init (env=0xffff77a668, argv=0xffff77a658, argc=1, l=<optimized out>) at dl-init.c:36
#4  _dl_init (main_map=0xffed9c3848, argc=<optimized out>, argv=0xffff77a658, env=0xffff77a668) at dl-init.c:93
#5  0x000000ffed9906e0 in _dl_start_user () from /lib64/ld.so.1
```

> Is it called from library constructor?

As per my observation, call to __pthread_initialize_minimal_internal () occurs even before preinit functions execute. 
And as fas as I know library constructors execute after preinit functions execute.

http://reviews.llvm.org/D6291

EMAIL PREFERENCES
  http://reviews.llvm.org/settings/panel/emailpreferences/