[Lldb-commits] [lldb] r255335 - Revert "Turn on new test summary results by default."

[Pavel Labath via lldb-commits]

The concept is not there to protect against timeouts, which are caused
by processes being too slow, for these we have been increasing
timeouts where necessary. This is there to guard against hangs in the
test suite (so the name might not be most fortunate, it would be
something like "excpected deadlock"), where increasing the timeout
does not help. E.g. TestEvents takes 4 seconds normally, but it does
deadlock quite often even with the 4m timeout, as you can see from the
redness. I have been trying to fix these, and the number are going
down, but I just don't have enough time to address them all (for now).

I don't find the concept per se that ridiculous, I think it is a good
idea to be able to mark any failure mode as "expected", to prevent
long red streaks on the buildbot (btw, I haven't checked, is it
possible to XFAIL crashes now?), although the current code that does
that is horrible.


On 11 December 2015 at 17:24, Todd Fiala <todd.fiala at gmail.com> wrote:
> (And by ridiculous, I mean the idea of having something time out, and having
> that be an expected condition just to keep the test red, is a heavy
> indication that something else needs to change --- feels like a bandaid on
> top of a bad patch job --- something I think we want to address at a more
> holistic level, which hopefully the low load test pass will help.  I think
> increasing the timeout would be a better way to handle that in the short
> term).
>
> On Fri, Dec 11, 2015 at 9:22 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
>>
>> I think this (the expected timeout) will go away once I'm done with the
>> low-load, single-worker follow up pass, fwiw.  If that is not the case, we
>> should be disabling tests that intermittently timeout for reasons other than
>> high load.
>>
>> On Fri, Dec 11, 2015 at 9:21 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
>>>
>>> "Expected timeout" - that concept is ridiculous.
>>>
>>> Why not increase the timeout?
>>>
>>> -Todd
>>>
>>> On Fri, Dec 11, 2015 at 3:06 AM, Pavel Labath via lldb-commits
>>> <lldb-commits at lists.llvm.org> wrote:
>>>>
>>>> Author: labath
>>>> Date: Fri Dec 11 05:05:24 2015
>>>> New Revision: 255335
>>>>
>>>> URL: http://llvm.org/viewvc/llvm-project?rev=255335&view=rev
>>>> Log:
>>>> Revert "Turn on new test summary results by default."
>>>>
>>>> The new test summary formatter does not honor the "expected timeout"
>>>> markings, which makes our
>>>> buildbots all red. I'm switching it off by default until we figure out a
>>>> way to make this work.
>>>>
>>>> Modified:
>>>>     lldb/trunk/packages/Python/lldbsuite/test/dotest.py
>>>>     lldb/trunk/packages/Python/lldbsuite/test/result_formatter.py
>>>>
>>>> Modified: lldb/trunk/packages/Python/lldbsuite/test/dotest.py
>>>> URL:
>>>> http://llvm.org/viewvc/llvm-project/lldb/trunk/packages/Python/lldbsuite/test/dotest.py?rev=255335&r1=255334&r2=255335&view=diff
>>>>
>>>> ==============================================================================
>>>> --- lldb/trunk/packages/Python/lldbsuite/test/dotest.py (original)
>>>> +++ lldb/trunk/packages/Python/lldbsuite/test/dotest.py Fri Dec 11
>>>> 05:05:24 2015
>>>> @@ -407,12 +407,6 @@ def parseOptionsAndInitTestdirs():
>>>>      if args.results_formatter_options:
>>>>          configuration.results_formatter_options =
>>>> args.results_formatter_options
>>>>
>>>> -    # Default to using the BasicResultsFormatter if no formatter is
>>>> specified
>>>> -    # and we're not a test inferior.
>>>> -    if not args.inferior and configuration.results_formatter_name is
>>>> None:
>>>> -        configuration.results_formatter_name = (
>>>> -
>>>> "lldbsuite.test.basic_results_formatter.BasicResultsFormatter")
>>>> -
>>>>      if args.lldb_platform_name:
>>>>          configuration.lldb_platform_name = args.lldb_platform_name
>>>>      if args.lldb_platform_url:
>>>>
>>>> Modified: lldb/trunk/packages/Python/lldbsuite/test/result_formatter.py
>>>> URL:
>>>> http://llvm.org/viewvc/llvm-project/lldb/trunk/packages/Python/lldbsuite/test/result_formatter.py?rev=255335&r1=255334&r2=255335&view=diff
>>>>
>>>> ==============================================================================
>>>> --- lldb/trunk/packages/Python/lldbsuite/test/result_formatter.py
>>>> (original)
>>>> +++ lldb/trunk/packages/Python/lldbsuite/test/result_formatter.py Fri
>>>> Dec 11 05:05:24 2015
>>>> @@ -683,9 +683,8 @@ class ResultsFormatter(object):
>>>>
>>>>              if event_type == "terminate":
>>>>                  self.terminate_called = True
>>>> -            elif event_type in [
>>>> -                    EventBuilder.TYPE_TEST_RESULT,
>>>> -                    EventBuilder.TYPE_JOB_RESULT]:
>>>> +            elif (event_type == EventBuilder.TYPE_TEST_RESULT or
>>>> +                    event_type == EventBuilder.TYPE_JOB_RESULT):
>>>>                  # Keep track of event counts per test/job result status
>>>> type.
>>>>                  # The only job (i.e. inferior process) results that
>>>> make it
>>>>                  # here are ones that cannot be remapped to the most
>>>> recently
>>>>
>>>>
>>>> _______________________________________________
>>>> lldb-commits mailing list
>>>> lldb-commits at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
>>>
>>>
>>>
>>>
>>> --
>>> -Todd
>>
>>
>>
>>
>> --
>> -Todd
>
>
>
>
> --
> -Todd