[Lldb-commits] [lldb] r255335 - Revert "Turn on new test summary results by default."

[Todd Fiala via lldb-commits]

(Sorry Pavel, missed your lldb-dev post on this.  See my comments there.
I'll pick it up there).

On Fri, Dec 11, 2015 at 9:36 AM, Pavel Labath <labath at google.com> wrote:

> The concept is not there to protect against timeouts, which are caused
> by processes being too slow, for these we have been increasing
> timeouts where necessary. This is there to guard against hangs in the
> test suite (so the name might not be most fortunate, it would be
> something like "excpected deadlock"), where increasing the timeout
> does not help. E.g. TestEvents takes 4 seconds normally, but it does
> deadlock quite often even with the 4m timeout, as you can see from the
> redness. I have been trying to fix these, and the number are going
> down, but I just don't have enough time to address them all (for now).
>
> I don't find the concept per se that ridiculous, I think it is a good
> idea to be able to mark any failure mode as "expected", to prevent
> long red streaks on the buildbot (btw, I haven't checked, is it
> possible to XFAIL crashes now?), although the current code that does
> that is horrible.
>
>
> On 11 December 2015 at 17:24, Todd Fiala <todd.fiala at gmail.com> wrote:
> > (And by ridiculous, I mean the idea of having something time out, and
> having
> > that be an expected condition just to keep the test red, is a heavy
> > indication that something else needs to change --- feels like a bandaid
> on
> > top of a bad patch job --- something I think we want to address at a more
> > holistic level, which hopefully the low load test pass will help.  I
> think
> > increasing the timeout would be a better way to handle that in the short
> > term).
> >
> > On Fri, Dec 11, 2015 at 9:22 AM, Todd Fiala <todd.fiala at gmail.com>
> wrote:
> >>
> >> I think this (the expected timeout) will go away once I'm done with the
> >> low-load, single-worker follow up pass, fwiw.  If that is not the case,
> we
> >> should be disabling tests that intermittently timeout for reasons other
> than
> >> high load.
> >>
> >> On Fri, Dec 11, 2015 at 9:21 AM, Todd Fiala <todd.fiala at gmail.com>
> wrote:
> >>>
> >>> "Expected timeout" - that concept is ridiculous.
> >>>
> >>> Why not increase the timeout?
> >>>
> >>> -Todd
> >>>
> >>> On Fri, Dec 11, 2015 at 3:06 AM, Pavel Labath via lldb-commits
> >>> <lldb-commits at lists.llvm.org> wrote:
> >>>>
> >>>> Author: labath
> >>>> Date: Fri Dec 11 05:05:24 2015
> >>>> New Revision: 255335
> >>>>
> >>>> URL: http://llvm.org/viewvc/llvm-project?rev=255335&view=rev
> >>>> Log:
> >>>> Revert "Turn on new test summary results by default."
> >>>>
> >>>> The new test summary formatter does not honor the "expected timeout"
> >>>> markings, which makes our
> >>>> buildbots all red. I'm switching it off by default until we figure
> out a
> >>>> way to make this work.
> >>>>
> >>>> Modified:
> >>>>     lldb/trunk/packages/Python/lldbsuite/test/dotest.py
> >>>>     lldb/trunk/packages/Python/lldbsuite/test/result_formatter.py
> >>>>
> >>>> Modified: lldb/trunk/packages/Python/lldbsuite/test/dotest.py
> >>>> URL:
> >>>>
> http://llvm.org/viewvc/llvm-project/lldb/trunk/packages/Python/lldbsuite/test/dotest.py?rev=255335&r1=255334&r2=255335&view=diff
> >>>>
> >>>>
> ==============================================================================
> >>>> --- lldb/trunk/packages/Python/lldbsuite/test/dotest.py (original)
> >>>> +++ lldb/trunk/packages/Python/lldbsuite/test/dotest.py Fri Dec 11
> >>>> 05:05:24 2015
> >>>> @@ -407,12 +407,6 @@ def parseOptionsAndInitTestdirs():
> >>>>      if args.results_formatter_options:
> >>>>          configuration.results_formatter_options =
> >>>> args.results_formatter_options
> >>>>
> >>>> -    # Default to using the BasicResultsFormatter if no formatter is
> >>>> specified
> >>>> -    # and we're not a test inferior.
> >>>> -    if not args.inferior and configuration.results_formatter_name is
> >>>> None:
> >>>> -        configuration.results_formatter_name = (
> >>>> -
> >>>> "lldbsuite.test.basic_results_formatter.BasicResultsFormatter")
> >>>> -
> >>>>      if args.lldb_platform_name:
> >>>>          configuration.lldb_platform_name = args.lldb_platform_name
> >>>>      if args.lldb_platform_url:
> >>>>
> >>>> Modified:
> lldb/trunk/packages/Python/lldbsuite/test/result_formatter.py
> >>>> URL:
> >>>>
> http://llvm.org/viewvc/llvm-project/lldb/trunk/packages/Python/lldbsuite/test/result_formatter.py?rev=255335&r1=255334&r2=255335&view=diff
> >>>>
> >>>>
> ==============================================================================
> >>>> --- lldb/trunk/packages/Python/lldbsuite/test/result_formatter.py
> >>>> (original)
> >>>> +++ lldb/trunk/packages/Python/lldbsuite/test/result_formatter.py Fri
> >>>> Dec 11 05:05:24 2015
> >>>> @@ -683,9 +683,8 @@ class ResultsFormatter(object):
> >>>>
> >>>>              if event_type == "terminate":
> >>>>                  self.terminate_called = True
> >>>> -            elif event_type in [
> >>>> -                    EventBuilder.TYPE_TEST_RESULT,
> >>>> -                    EventBuilder.TYPE_JOB_RESULT]:
> >>>> +            elif (event_type == EventBuilder.TYPE_TEST_RESULT or
> >>>> +                    event_type == EventBuilder.TYPE_JOB_RESULT):
> >>>>                  # Keep track of event counts per test/job result
> status
> >>>> type.
> >>>>                  # The only job (i.e. inferior process) results that
> >>>> make it
> >>>>                  # here are ones that cannot be remapped to the most
> >>>> recently
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> lldb-commits mailing list
> >>>> lldb-commits at lists.llvm.org
> >>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> -Todd
> >>
> >>
> >>
> >>
> >> --
> >> -Todd
> >
> >
> >
> >
> > --
> > -Todd
>



-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-commits/attachments/20151211/8a60f3ae/attachment-0001.html>