[lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

These went in as:

r255130 - turn it on by default
r255131 - create known issues.  This one is to be reverted if all 3 types
show up properly.

On Wed, Dec 9, 2015 at 9:41 AM, Todd Fiala <todd.fiala at gmail.com> wrote:

> It is a small change.
>
> I almost have all the trial tests ready, so I'll just commit both changes
> at the same time (the flip on, and the trial balloon issues).
>
> If all goes well and the three types of issue show up, then the last of
> the two will get reverted (the one with the failures).
>
> If none (or only some) of the issues show up, they'll both get reverted.
>
> -Todd
>
> On Wed, Dec 9, 2015 at 9:39 AM, Pavel Labath <labath at google.com> wrote:
>
>> If it's not too much work, I think the extra bit of noise will not be
>> a problem. But I don't think it is really necessary either.
>>
>> I assume the actual flip will be a small change that we can back out
>> easily if we notice troubles... After a sufficient grace period we can
>> remove the old formatter altogether and hopefully simplify the code
>> somewhat.
>>
>> pl
>>
>> On 9 December 2015 at 17:08, Todd Fiala <todd.fiala at gmail.com> wrote:
>> > Here's what I can do.
>> >
>> > Put in the change (setting the default to use the new format).
>> >
>> > Separately, put in a trial balloon commit with one failing test, one
>> > exceptional exit test, and one timeout test, and watch the ubuntu 14.04
>> > buildbot catch it and fail.  Then reverse this out.  That should show
>> beyond
>> > a reasonable doubt whether the buildbot catches new failures and
>> errors.  (I
>> > think this is a noisy way to accomplish this, but it certainly would
>> > validate if its working).
>> >
>> > -Todd
>> >
>> > On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala <todd.fiala at gmail.com>
>> wrote:
>> >>
>> >> Specifically, the markers for issue details are:
>> >>
>> >> FAIL
>> >> ERROR
>> >> UNEXPECTED SUCCESS
>> >> TIMEOUT
>> >>
>> >> (These are the fourth field in the array entries (lines 275 - 290) of
>> >> packages/Python/lldbsuite/test/basic_results_formatter.py).
>> >>
>> >> -Todd
>> >>
>> >> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala <todd.fiala at gmail.com>
>> wrote:
>> >>>
>> >>> That's a good point, Tamas.
>> >>>
>> >>> I use (so I claim) the same all upper-case markers for the test result
>> >>> details.  Including, not using XPASS but rather UNEXPECTED SUCCESS for
>> >>> unexpected successes.  (The former would trigger the lit script IIRC
>> to
>> >>> parse that as a failing-style result).
>> >>>
>> >>> The intent is this is a no-op on the test runner.
>> >>>
>> >>> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer <
>> tberghammer at google.com>
>> >>> wrote:
>> >>>>
>> >>>> +Ying Chen
>> >>>>
>> >>>> Ying, what do we have to do on the build bot side to support a
>> change in
>> >>>> the default test result summary formatter?
>> >>>>
>> >>>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev
>> >>>> <lldb-dev at lists.llvm.org> wrote:
>> >>>>>
>> >>>>> Hi all,
>> >>>>>
>> >>>>> Per a previous thread on this, I've made all the changes I intended
>> to
>> >>>>> make last night to get the intended replacement of test run results
>> meet or
>> >>>>> exceed current requirements.
>> >>>>>
>> >>>>> I'd like to switch over to that by default.  I'm depending on the
>> test
>> >>>>> event system to be able to handle test method reruns in test results
>> >>>>> accounting.
>> >>>>>
>> >>>>> The primary thing missing before was that timeouts were not routed
>> >>>>> through the test events system, nor were exception process exits
>> (i.e. test
>> >>>>> inferiors exiting with a signal on POSIX systems).  Those were
>> added last
>> >>>>> night so that test events are generated for those, and the
>> >>>>> BasicResultsFormatter presents that information properly.
>> >>>>>
>> >>>>> I will switch it over to being the default output in a bit here.
>> >>>>> Please let me know if you have any concerns once I flip it on by
>> default.
>> >>>>>
>> >>>>> Thanks!
>> >>>>> --
>> >>>>> -Todd
>> >>>>> _______________________________________________
>> >>>>> lldb-dev mailing list
>> >>>>> lldb-dev at lists.llvm.org
>> >>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> -Todd
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> -Todd
>> >
>> >
>> >
>> >
>> > --
>> > -Todd
>>
>
>
>
> --
> -Todd
>



-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151209/c43bbcab/attachment-0001.html>