[lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

I forced a build on the ubuntu 14.04 cmake builder.  The build _after_ 9292
will contain the two changes (and we will expect failures on it).

On Wed, Dec 9, 2015 at 10:05 AM, Todd Fiala <todd.fiala at gmail.com> wrote:

> These went in as:
>
> r255130 - turn it on by default
> r255131 - create known issues.  This one is to be reverted if all 3 types
> show up properly.
>
> On Wed, Dec 9, 2015 at 9:41 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> It is a small change.
>>
>> I almost have all the trial tests ready, so I'll just commit both changes
>> at the same time (the flip on, and the trial balloon issues).
>>
>> If all goes well and the three types of issue show up, then the last of
>> the two will get reverted (the one with the failures).
>>
>> If none (or only some) of the issues show up, they'll both get reverted.
>>
>> -Todd
>>
>> On Wed, Dec 9, 2015 at 9:39 AM, Pavel Labath <labath at google.com> wrote:
>>
>>> If it's not too much work, I think the extra bit of noise will not be
>>> a problem. But I don't think it is really necessary either.
>>>
>>> I assume the actual flip will be a small change that we can back out
>>> easily if we notice troubles... After a sufficient grace period we can
>>> remove the old formatter altogether and hopefully simplify the code
>>> somewhat.
>>>
>>> pl
>>>
>>> On 9 December 2015 at 17:08, Todd Fiala <todd.fiala at gmail.com> wrote:
>>> > Here's what I can do.
>>> >
>>> > Put in the change (setting the default to use the new format).
>>> >
>>> > Separately, put in a trial balloon commit with one failing test, one
>>> > exceptional exit test, and one timeout test, and watch the ubuntu 14.04
>>> > buildbot catch it and fail.  Then reverse this out.  That should show
>>> beyond
>>> > a reasonable doubt whether the buildbot catches new failures and
>>> errors.  (I
>>> > think this is a noisy way to accomplish this, but it certainly would
>>> > validate if its working).
>>> >
>>> > -Todd
>>> >
>>> > On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala <todd.fiala at gmail.com>
>>> wrote:
>>> >>
>>> >> Specifically, the markers for issue details are:
>>> >>
>>> >> FAIL
>>> >> ERROR
>>> >> UNEXPECTED SUCCESS
>>> >> TIMEOUT
>>> >>
>>> >> (These are the fourth field in the array entries (lines 275 - 290) of
>>> >> packages/Python/lldbsuite/test/basic_results_formatter.py).
>>> >>
>>> >> -Todd
>>> >>
>>> >> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala <todd.fiala at gmail.com>
>>> wrote:
>>> >>>
>>> >>> That's a good point, Tamas.
>>> >>>
>>> >>> I use (so I claim) the same all upper-case markers for the test
>>> result
>>> >>> details.  Including, not using XPASS but rather UNEXPECTED SUCCESS
>>> for
>>> >>> unexpected successes.  (The former would trigger the lit script IIRC
>>> to
>>> >>> parse that as a failing-style result).
>>> >>>
>>> >>> The intent is this is a no-op on the test runner.
>>> >>>
>>> >>> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer <
>>> tberghammer at google.com>
>>> >>> wrote:
>>> >>>>
>>> >>>> +Ying Chen
>>> >>>>
>>> >>>> Ying, what do we have to do on the build bot side to support a
>>> change in
>>> >>>> the default test result summary formatter?
>>> >>>>
>>> >>>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev
>>> >>>> <lldb-dev at lists.llvm.org> wrote:
>>> >>>>>
>>> >>>>> Hi all,
>>> >>>>>
>>> >>>>> Per a previous thread on this, I've made all the changes I
>>> intended to
>>> >>>>> make last night to get the intended replacement of test run
>>> results meet or
>>> >>>>> exceed current requirements.
>>> >>>>>
>>> >>>>> I'd like to switch over to that by default.  I'm depending on the
>>> test
>>> >>>>> event system to be able to handle test method reruns in test
>>> results
>>> >>>>> accounting.
>>> >>>>>
>>> >>>>> The primary thing missing before was that timeouts were not routed
>>> >>>>> through the test events system, nor were exception process exits
>>> (i.e. test
>>> >>>>> inferiors exiting with a signal on POSIX systems).  Those were
>>> added last
>>> >>>>> night so that test events are generated for those, and the
>>> >>>>> BasicResultsFormatter presents that information properly.
>>> >>>>>
>>> >>>>> I will switch it over to being the default output in a bit here.
>>> >>>>> Please let me know if you have any concerns once I flip it on by
>>> default.
>>> >>>>>
>>> >>>>> Thanks!
>>> >>>>> --
>>> >>>>> -Todd
>>> >>>>> _______________________________________________
>>> >>>>> lldb-dev mailing list
>>> >>>>> lldb-dev at lists.llvm.org
>>> >>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> -Todd
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> -Todd
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > -Todd
>>>
>>
>>
>>
>> --
>> -Todd
>>
>
>
>
> --
> -Todd
>



-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151209/29ff3a7d/attachment.html>