[lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

It is a small change.

I almost have all the trial tests ready, so I'll just commit both changes
at the same time (the flip on, and the trial balloon issues).

If all goes well and the three types of issue show up, then the last of the
two will get reverted (the one with the failures).

If none (or only some) of the issues show up, they'll both get reverted.

-Todd

On Wed, Dec 9, 2015 at 9:39 AM, Pavel Labath <labath at google.com> wrote:

> If it's not too much work, I think the extra bit of noise will not be
> a problem. But I don't think it is really necessary either.
>
> I assume the actual flip will be a small change that we can back out
> easily if we notice troubles... After a sufficient grace period we can
> remove the old formatter altogether and hopefully simplify the code
> somewhat.
>
> pl
>
> On 9 December 2015 at 17:08, Todd Fiala <todd.fiala at gmail.com> wrote:
> > Here's what I can do.
> >
> > Put in the change (setting the default to use the new format).
> >
> > Separately, put in a trial balloon commit with one failing test, one
> > exceptional exit test, and one timeout test, and watch the ubuntu 14.04
> > buildbot catch it and fail.  Then reverse this out.  That should show
> beyond
> > a reasonable doubt whether the buildbot catches new failures and
> errors.  (I
> > think this is a noisy way to accomplish this, but it certainly would
> > validate if its working).
> >
> > -Todd
> >
> > On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
> >>
> >> Specifically, the markers for issue details are:
> >>
> >> FAIL
> >> ERROR
> >> UNEXPECTED SUCCESS
> >> TIMEOUT
> >>
> >> (These are the fourth field in the array entries (lines 275 - 290) of
> >> packages/Python/lldbsuite/test/basic_results_formatter.py).
> >>
> >> -Todd
> >>
> >> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala <todd.fiala at gmail.com>
> wrote:
> >>>
> >>> That's a good point, Tamas.
> >>>
> >>> I use (so I claim) the same all upper-case markers for the test result
> >>> details.  Including, not using XPASS but rather UNEXPECTED SUCCESS for
> >>> unexpected successes.  (The former would trigger the lit script IIRC to
> >>> parse that as a failing-style result).
> >>>
> >>> The intent is this is a no-op on the test runner.
> >>>
> >>> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer <
> tberghammer at google.com>
> >>> wrote:
> >>>>
> >>>> +Ying Chen
> >>>>
> >>>> Ying, what do we have to do on the build bot side to support a change
> in
> >>>> the default test result summary formatter?
> >>>>
> >>>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev
> >>>> <lldb-dev at lists.llvm.org> wrote:
> >>>>>
> >>>>> Hi all,
> >>>>>
> >>>>> Per a previous thread on this, I've made all the changes I intended
> to
> >>>>> make last night to get the intended replacement of test run results
> meet or
> >>>>> exceed current requirements.
> >>>>>
> >>>>> I'd like to switch over to that by default.  I'm depending on the
> test
> >>>>> event system to be able to handle test method reruns in test results
> >>>>> accounting.
> >>>>>
> >>>>> The primary thing missing before was that timeouts were not routed
> >>>>> through the test events system, nor were exception process exits
> (i.e. test
> >>>>> inferiors exiting with a signal on POSIX systems).  Those were added
> last
> >>>>> night so that test events are generated for those, and the
> >>>>> BasicResultsFormatter presents that information properly.
> >>>>>
> >>>>> I will switch it over to being the default output in a bit here.
> >>>>> Please let me know if you have any concerns once I flip it on by
> default.
> >>>>>
> >>>>> Thanks!
> >>>>> --
> >>>>> -Todd
> >>>>> _______________________________________________
> >>>>> lldb-dev mailing list
> >>>>> lldb-dev at lists.llvm.org
> >>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> -Todd
> >>
> >>
> >>
> >>
> >> --
> >> -Todd
> >
> >
> >
> >
> > --
> > -Todd
>



-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151209/e384703c/attachment.html>