[compiler-rt] r315777 - [asan] Deflake one test by running it 3 times.

Tue Oct 17 13:01:34 PDT 2017

I'm sure it's supported. Relevant platform sources has changed a lot
recently, even this message is gone from the source, so maybe the
error can be fixed by updating. I don't know. Debugging flaky tests
takes a lot of time and I'm pretty sure that in this case the bug is
not on our side, so it's hard to justify the investment. I've reverted
the change in r316023 and disabled the test on (android && i386).

On Mon, Oct 16, 2017 at 6:56 PM, David Blaikie <dblaikie at gmail.com> wrote:
> But if it's supported, is this a bug that's going to be fixed? (on either
> side - platform or compiler) - or is this functionality unsupported on this
> platform? (if it's flaky, it's hard to say it's supported, that's not a good
> user experience)
>
> On Mon, Oct 16, 2017 at 4:27 PM Evgenii Stepanov <eugeni.stepanov at gmail.com>
> wrote:
>>
>> Well, it's an x86 device, so a bit of an oddity, but it is still
>> supported. It runs an OS release from 2016 (N). It's a reasonable
>> configuration.
>>
>>
>> On Mon, Oct 16, 2017 at 2:42 PM, David Blaikie <dblaikie at gmail.com> wrote:
>> > Could the device be removed from testing? (is it likely the device
>> > that's at
>> > fault, more than the test - or is it a truly representative, valid
>> > device
>> > where any device in that configuration would have this problem and it's
>> > a
>> > supported configuration (not too old, etc)?)
>> >
>> > On Mon, Oct 16, 2017 at 2:38 PM Evgenii Stepanov via llvm-commits
>> > <llvm-commits at lists.llvm.org> wrote:
>> >>
>> >> For some value of x, p^x is low enough :)
>> >>
>> >> Seriously speaking, the test is failing on a particular device and I
>> >> suspect it could be a kernel issue: it is running 3.10, which is quite
>> >> old. I don't have time to debug it, but I could make the test as
>> >> failing on (android && i686) which is not the real cause of the error,
>> >> but it would make the bot green and keep the test exercised on the
>> >> arm-android device.
>> >>
>> >> On Mon, Oct 16, 2017 at 2:09 PM, Rui Ueyama via llvm-commits
>> >> <llvm-commits at lists.llvm.org> wrote:
>> >> > You cannot fix a flakey test by just running it more than once, can
>> >> > you?
>> >> > I
>> >> > mean, the probability of test failure decreases from p to p^3, but it
>> >> > is
>> >> > still not zero.
>> >> >
>> >> > On Mon, Oct 16, 2017 at 2:03 PM, Vitaly Buka via llvm-commits
>> >> > <llvm-commits at lists.llvm.org> wrote:
>> >> >>
>> >> >> It does not makes test slower as it passes on all platforms from the
>> >> >> first
>> >> >> time.
>> >> >>
>> >> >> We don't suspect issues with functionally, rather some specific
>> >> >> platform
>> >> >> bug.
>> >> >> Also on Android we know only one device were it fails. And I guess
>> >> >> it
>> >> >> fails only in Asan mode?
>> >> >> So if we mark it unsupported, it should be very specific.
>> >> >> E.g. // UNSUPPORTED: android && asan && x86 (not sure if that one is
>> >> >> 64bit
>> >> >> and how we exactly recognize x86 on android)
>> >> >>
>> >> >> On Mon, Oct 16, 2017 at 1:49 PM, David Blaikie via llvm-commits
>> >> >> <llvm-commits at lists.llvm.org> wrote:
>> >> >>>
>> >> >>> If this functionality isn't reliable on Android, I think
>> >> >>> UNSUPPORTED/XFAIL is more appropriate than deflaking like this. The
>> >> >>> functionality's no good if it fails that much, so why test/support
>> >> >>> it,
>> >> >>> making tests slower and risk flakiness creeping in on other
>> >> >>> platforms
>> >> >>> due to
>> >> >>> the more generous testing?
>> >> >>>
>> >> >>> On Fri, Oct 13, 2017 at 4:57 PM Evgeniy Stepanov via llvm-commits
>> >> >>> <llvm-commits at lists.llvm.org> wrote:
>> >> >>>>
>> >> >>>> Author: eugenis
>> >> >>>> Date: Fri Oct 13 16:57:08 2017
>> >> >>>> New Revision: 315777
>> >> >>>>
>> >> >>>> URL: http://llvm.org/viewvc/llvm-project?rev=315777&view=rev
>> >> >>>> Log:
>> >> >>>> [asan] Deflake one test by running it 3 times.
>> >> >>>>
>> >> >>>> The test seems to trigger an android platform bug under load.
>> >> >>>>
>> >> >>>> Modified:
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> compiler-rt/trunk/test/sanitizer_common/TestCases/Linux/allow_user_segv.cc
>> >> >>>>
>> >> >>>> Modified:
>> >> >>>>
>> >> >>>>
>> >> >>>> compiler-rt/trunk/test/sanitizer_common/TestCases/Linux/allow_user_segv.cc
>> >> >>>> URL:
>> >> >>>>
>> >> >>>>
>> >> >>>> http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/sanitizer_common/TestCases/Linux/allow_user_segv.cc?rev=315777&r1=315776&r2=315777&view=diff
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> ==============================================================================
>> >> >>>> ---
>> >> >>>>
>> >> >>>>
>> >> >>>> compiler-rt/trunk/test/sanitizer_common/TestCases/Linux/allow_user_segv.cc
>> >> >>>> (original)
>> >> >>>> +++
>> >> >>>>
>> >> >>>>
>> >> >>>> compiler-rt/trunk/test/sanitizer_common/TestCases/Linux/allow_user_segv.cc
>> >> >>>> Fri Oct 13 16:57:08 2017
>> >> >>>> @@ -4,16 +4,34 @@
>> >> >>>>  // clang-format off
>> >> >>>>  // RUN: %clangxx -O0 %s -o %t
>> >> >>>>
>> >> >>>> +// RUN: %env_tool_opts=handle_segv=0 not %run %t 2>&1 | FileCheck
>> >> >>>> %s
>> >> >>>> --check-prefix=CHECK0 || \
>> >> >>>> +// RUN: %env_tool_opts=handle_segv=0 not %run %t 2>&1 | FileCheck
>> >> >>>> %s
>> >> >>>> --check-prefix=CHECK0 || \
>> >> >>>>  // RUN: %env_tool_opts=handle_segv=0 not %run %t 2>&1 | FileCheck
>> >> >>>> %s
>> >> >>>> --check-prefix=CHECK0
>> >> >>>> +// RUN: %env_tool_opts=handle_segv=1 not %run %t 2>&1 | FileCheck
>> >> >>>> %s
>> >> >>>> --check-prefix=CHECK1 || \
>> >> >>>> +// RUN: %env_tool_opts=handle_segv=1 not %run %t 2>&1 | FileCheck
>> >> >>>> %s
>> >> >>>> --check-prefix=CHECK1 || \
>> >> >>>>  // RUN: %env_tool_opts=handle_segv=1 not %run %t 2>&1 | FileCheck
>> >> >>>> %s
>> >> >>>> --check-prefix=CHECK1
>> >> >>>> +// RUN: %env_tool_opts=handle_segv=2 not %run %t 2>&1 | FileCheck
>> >> >>>> %s
>> >> >>>> --check-prefix=CHECK2 || \
>> >> >>>> +// RUN: %env_tool_opts=handle_segv=2 not %run %t 2>&1 | FileCheck
>> >> >>>> %s
>> >> >>>> --check-prefix=CHECK2 || \
>> >> >>>>  // RUN: %env_tool_opts=handle_segv=2 not %run %t 2>&1 | FileCheck
>> >> >>>> %s
>> >> >>>> --check-prefix=CHECK2
>> >> >>>>
>> >> >>>> +// RUN: %env_tool_opts=handle_segv=0:allow_user_segv_handler=0
>> >> >>>> not
>> >> >>>> %run
>> >> >>>> %t 2>&1 | FileCheck %s --check-prefix=CHECK0 || \
>> >> >>>> +// RUN: %env_tool_opts=handle_segv=0:allow_user_segv_handler=0
>> >> >>>> not
>> >> >>>> %run
>> >> >>>> %t 2>&1 | FileCheck %s --check-prefix=CHECK0 || \
>> >> >>>>  // RUN: %env_tool_opts=handle_segv=0:allow_user_segv_handler=0
>> >> >>>> not
>> >> >>>> %run
>> >> >>>> %t 2>&1 | FileCheck %s --check-prefix=CHECK0
>> >> >>>> +// RUN: %env_tool_opts=handle_segv=1:allow_user_segv_handler=0
>> >> >>>> not
>> >> >>>> %run
>> >> >>>> %t 2>&1 | FileCheck %s --check-prefix=CHECK2 || \
>> >> >>>> +// RUN: %env_tool_opts=handle_segv=1:allow_user_segv_handler=0
>> >> >>>> not
>> >> >>>> %run
>> >> >>>> %t 2>&1 | FileCheck %s --check-prefix=CHECK2 || \
>> >> >>>>  // RUN: %env_tool_opts=handle_segv=1:allow_user_segv_handler=0
>> >> >>>> not
>> >> >>>> %run
>> >> >>>> %t 2>&1 | FileCheck %s --check-prefix=CHECK2
>> >> >>>> +// RUN: %env_tool_opts=handle_segv=2:allow_user_segv_handler=0
>> >> >>>> not
>> >> >>>> %run
>> >> >>>> %t 2>&1 | FileCheck %s --check-prefix=CHECK2 || \
>> >> >>>> +// RUN: %env_tool_opts=handle_segv=2:allow_user_segv_handler=0
>> >> >>>> not
>> >> >>>> %run
>> >> >>>> %t 2>&1 | FileCheck %s --check-prefix=CHECK2 || \
>> >> >>>>  // RUN: %env_tool_opts=handle_segv=2:allow_user_segv_handler=0
>> >> >>>> not
>> >> >>>> %run
>> >> >>>> %t 2>&1 | FileCheck %s --check-prefix=CHECK2
>> >> >>>>
>> >> >>>> +// RUN: %env_tool_opts=handle_segv=0:allow_user_segv_handler=1
>> >> >>>> not
>> >> >>>> %run
>> >> >>>> %t 2>&1 | FileCheck %s --check-prefix=CHECK0 || \
>> >> >>>> +// RUN: %env_tool_opts=handle_segv=0:allow_user_segv_handler=1
>> >> >>>> not
>> >> >>>> %run
>> >> >>>> %t 2>&1 | FileCheck %s --check-prefix=CHECK0 || \
>> >> >>>>  // RUN: %env_tool_opts=handle_segv=0:allow_user_segv_handler=1
>> >> >>>> not
>> >> >>>> %run
>> >> >>>> %t 2>&1 | FileCheck %s --check-prefix=CHECK0
>> >> >>>> +// RUN: %env_tool_opts=handle_segv=1:allow_user_segv_handler=1
>> >> >>>> not
>> >> >>>> %run
>> >> >>>> %t 2>&1 | FileCheck %s --check-prefix=CHECK1 || \
>> >> >>>> +// RUN: %env_tool_opts=handle_segv=1:allow_user_segv_handler=1
>> >> >>>> not
>> >> >>>> %run
>> >> >>>> %t 2>&1 | FileCheck %s --check-prefix=CHECK1 || \
>> >> >>>>  // RUN: %env_tool_opts=handle_segv=1:allow_user_segv_handler=1
>> >> >>>> not
>> >> >>>> %run
>> >> >>>> %t 2>&1 | FileCheck %s --check-prefix=CHECK1
>> >> >>>> +// RUN: %env_tool_opts=handle_segv=2:allow_user_segv_handler=1
>> >> >>>> not
>> >> >>>> %run
>> >> >>>> %t 2>&1 | FileCheck %s --check-prefix=CHECK2 || \
>> >> >>>> +// RUN: %env_tool_opts=handle_segv=2:allow_user_segv_handler=1
>> >> >>>> not
>> >> >>>> %run
>> >> >>>> %t 2>&1 | FileCheck %s --check-prefix=CHECK2 || \
>> >> >>>>  // RUN: %env_tool_opts=handle_segv=2:allow_user_segv_handler=1
>> >> >>>> not
>> >> >>>> %run
>> >> >>>> %t 2>&1 | FileCheck %s --check-prefix=CHECK2
>> >> >>>>  // clang-format on
>> >> >>>>
>> >> >>>> @@ -21,9 +39,6 @@
>> >> >>>>  // XFAIL: msan
>> >> >>>>  // XFAIL: tsan
>> >> >>>>
>> >> >>>> -// Flaky errors in debuggerd with "waitpid returned unexpected
>> >> >>>> pid
>> >> >>>> (0)"
>> >> >>>> in logcat.
>> >> >>>> -// UNSUPPORTED: android
>> >> >>>> -
>> >> >>>>  #include <signal.h>
>> >> >>>>  #include <stdio.h>
>> >> >>>>  #include <stdlib.h>
>> >> >>>>
>> >> >>>>
>> >> >>>> _______________________________________________
>> >> >>>> llvm-commits mailing list
>> >> >>>> llvm-commits at lists.llvm.org
>> >> >>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>> >> >>>
>> >> >>>
>> >> >>> _______________________________________________
>> >> >>> llvm-commits mailing list
>> >> >>> llvm-commits at lists.llvm.org
>> >> >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>> >> >>>
>> >> >>
>> >> >>
>> >> >> _______________________________________________
>> >> >> llvm-commits mailing list
>> >> >> llvm-commits at lists.llvm.org
>> >> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>> >> >>
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > llvm-commits mailing list
>> >> > llvm-commits at lists.llvm.org
>> >> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>> >> >
>> >> _______________________________________________
>> >> llvm-commits mailing list
>> >> llvm-commits at lists.llvm.org
>> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits