[compiler-rt] r275539 - [asan] Reduce flakiness of halt_on_error-torture.cc testcase.

Tue Jul 26 13:24:41 PDT 2016

On Tue, Jul 26, 2016 at 6:09 PM, Maxim Ostapenko
<m.ostapenko at samsung.com> wrote:
> On 26/07/16 11:45, Maxim Ostapenko wrote:
>>
>> On 22/07/16 04:13, Kostya Serebryany via llvm-commits wrote:
>>>
>>> Do you think this is fully fixed?
>>> I've seen it fail on my local machine just now (passes when I repeat)
>>
>>
>> Oh, sorry, I've missed this mail (that nasty corporate mail changing).
>> I'll take a look on this.
>
>
> /home/max/build/llvm/./bin/clang --driver-mode=g++ -fsanitize=address
> -mno-omit-leaf-frame-pointer -fno-omit-frame-pointer
> -fno-optimize-sibling-calls -gline-tables-only -m64
> -fsanitize-recover=address -pthread
> /home/max/src/llvm/projects/compiler-rt/test/asan/TestCases/Posix/halt_on_error-torture.cc
> -o
> /home/max/build/llvm/projects/compiler-rt/test/asan/X86_64LinuxConfig/TestCases/Posix/Output/halt_on_error-torture.cc.tmp
> env ASAN_OPTIONS=halt_on_error=false:suppress_equal_pcs=false
> /home/max/build/llvm/projects/compiler-rt/test/asan/X86_64LinuxConfig/TestCases/Posix/Output/halt_on_error-torture.cc.tmp
> 10 20 >10.txt 2>&1 || true
>
> FileCheck --check-prefix=CHECK-COLLISION
> /home/max/src/llvm/projects/compiler-rt/test/asan/TestCases/Posix/halt_on_error-torture.cc
> < 10.txt || FileCheck --check-prefix=CHECK-NO-COLLISION
> /home/max/src/llvm/projects/compiler-rt/test/asan/TestCases/Posix/halt_on_error-torture.cc
> < 10.txt
>
> --
> Exit Code: 1
>
> Command Output (stderr):
> --
> /home/max/src/llvm/projects/compiler-rt/test/asan/TestCases/Posix/halt_on_error-torture.cc:76:22:
> error: expected string not found in input
>  // CHECK-COLLISION: AddressSanitizer: nested bug in the same thread,
> aborting
>                      ^
> <stdin>:1:1: note: scanning from here
> =================================================================
> ^
> <stdin>:22:10: note: possible intended match here
> SUMMARY: AddressSanitizer: use-after-poison
> /home/max/src/llvm/projects/compiler-rt/test/asan/TestCases/Posix/halt_on_error-torture.cc:45:14
> in run(void*)
>          ^
> /home/max/src/llvm/projects/compiler-rt/test/asan/TestCases/Posix/halt_on_error-torture.cc:77:25:
> error: expected string not found in input
>  // CHECK-NO-COLLISION: All threads terminated
>                         ^
> <stdin>:1:1: note: scanning from here
> =================================================================
> ^
> <stdin>:3:31: note: possible intended match here
> WRITE of size 1 at 0x7f36b00fedb0 thread T1
>                               ^
>
> --
>
> ********************
>
> Here what I found out:
> When we run halt_on_error-torture.cc with 10 threads and 20 iterations with
> *halt_on_error=false:suppress_equal_pcs=false*, we write 200 reports to
> 10.txt file and sometimes have collisions. We have CHECK-COLLISION check
> that greps 'AddressSanitizer: nested bug in the same thread, aborting'
> message in 10.txt, but for some reason it doesn't contain this line. If I
> don't redirect stderr > 10.txt 'AddressSanitizer: nested bug in the same
> thread, aborting' is printed to my screen as expected.
>
> I suspect that we hit on race in WriteToFile function called from
> ScopedInErrorReport constructor:

If that is the case then it's a bug as only one thread should get into
this section. But frankly I do not see any proofs.

>     u32 current_tid = GetCurrentTidOrInvalid();
>     if (reporting_thread_tid_ == current_tid ||
>         reporting_thread_tid_ == kInvalidTid) {
>       // This is either asynch signal or nested error during error
> reporting.
>       // Fail simple to avoid deadlocks in Report().
>
>       // Can't use Report() here because of potential deadlocks
>       // in nested signal handlers.
>       const char msg[] = "AddressSanitizer: nested bug in the same thread, "
>                          "aborting.\n";
>       WriteToFile(kStderrFd, msg, sizeof(msg));
>
>       internal__exit(common_flags()->exitcode);
>     }
>
> Here we get reporting_thread_tid_ == kInvalidTid and trying to print msg to
> kStderrFd while another thread tries to print its report.
>
> I wonder how can we fix this? Perhaps we could just drop flaky check in
> halt_on_error-torture.cc testcase?

I suggest we don't drop anything until we understand the core reasons
and fix bugs if any. If test indeed proves to contain unavoidable
flakiness (which I doubt), then yes, we'll have to disable it.

> -Maxim
>
>
>>
>>>
>>> On Fri, Jul 15, 2016 at 12:57 AM, Maxim Ostapenko via llvm-commits
>>> <llvm-commits at lists.llvm.org <mailto:llvm-commits at lists.llvm.org>> wrote:
>>>
>>>     Author: chefmax
>>>     Date: Fri Jul 15 02:57:43 2016
>>>     New Revision: 275539
>>>
>>>     URL: http://llvm.org/viewvc/llvm-project?rev=275539&view=rev
>>>     Log:
>>>     [asan] Reduce flakiness of halt_on_error-torture.cc testcase.
>>>
>>>     It seems in some situations we have clashes on very first error so
>>>     test only prints "nested bug in the same thread, aborting" rather
>>>     than "use-after-poison", so remove corresponding " RUN: FileCheck
>>>     %s < 10.txt" line.
>>>     Also, the two last " RUN: FileCheck --check-prefix=CHECK-COLLISION
>>>     %s < 1.txt || FileCheck --check-prefix=CHECK-NO-COLLISION %s <
>>>     1.txt" look wrong, they should check 10.txt. Fix these lines too.
>>>
>>>     Differential Revision: https://reviews.llvm.org/D22309
>>>
>>>     Modified:
>>> compiler-rt/trunk/test/asan/TestCases/Posix/halt_on_error-torture.cc
>>>
>>>     Modified:
>>> compiler-rt/trunk/test/asan/TestCases/Posix/halt_on_error-torture.cc
>>>     URL:
>>>
>>> http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/asan/TestCases/Posix/halt_on_error-torture.cc?rev=275539&r1=275538&r2=275539&view=diff
>>>
>>> ==============================================================================
>>>     ---
>>> compiler-rt/trunk/test/asan/TestCases/Posix/halt_on_error-torture.cc
>>>     (original)
>>>     +++
>>> compiler-rt/trunk/test/asan/TestCases/Posix/halt_on_error-torture.cc
>>>     Fri Jul 15 02:57:43 2016
>>>     @@ -9,15 +9,11 @@
>>>      //
>>>      // Collisions are unlikely but still possible so we need the ||.
>>>      // RUN:
>>>     %env_asan_opts=halt_on_error=false:suppress_equal_pcs=false %run
>>>     %t 10 20 >10.txt 2>&1 || true
>>>     -// This one is racy although _very_ unlikely to fail:
>>>     -// RUN: FileCheck %s < 10.txt
>>>     -// RUN: FileCheck --check-prefix=CHECK-COLLISION %s < 1.txt ||
>>>     FileCheck --check-prefix=CHECK-NO-COLLISION %s < 1.txt
>>>     +// RUN: FileCheck --check-prefix=CHECK-COLLISION %s < 10.txt ||
>>>     FileCheck --check-prefix=CHECK-NO-COLLISION %s < 10.txt
>>>      //
>>>      // Collisions are unlikely but still possible so we need the ||.
>>>      // RUN: %env_asan_opts=halt_on_error=false %run %t 10 20 >10.txt
>>>     2>&1 || true
>>>     -// This one is racy although _very_ unlikely to fail:
>>>     -// RUN: FileCheck %s < 10.txt
>>>     -// RUN: FileCheck --check-prefix=CHECK-COLLISION %s < 1.txt ||
>>>     FileCheck --check-prefix=CHECK-NO-COLLISION %s < 1.txt
>>>     +// RUN: FileCheck --check-prefix=CHECK-COLLISION %s < 10.txt ||
>>>     FileCheck --check-prefix=CHECK-NO-COLLISION %s < 10.txt
>>>
>>>      #include <stdio.h>
>>>      #include <stdlib.h>
>>>
>>>
>>>     _______________________________________________
>>>     llvm-commits mailing list
>>>     llvm-commits at lists.llvm.org <mailto:llvm-commits at lists.llvm.org>
>>>     http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> llvm-commits mailing list
>>> llvm-commits at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>
>>
>