[lldb-dev] BasicResultsFormatter - new test results summary

Wed Dec 9 10:13:47 PST 2015

I am going to stop the current build on that builder.  There was one change
in it, and it will be another 20 minutes before it completes.  I don't want
the repo in a known broken state that long.

On Wed, Dec 9, 2015 at 10:07 AM, Todd Fiala <todd.fiala at gmail.com> wrote:

> I forced a build on the ubuntu 14.04 cmake builder.  The build _after_
> 9292 will contain the two changes (and we will expect failures on it).
>
> On Wed, Dec 9, 2015 at 10:05 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> These went in as:
>>
>> r255130 - turn it on by default
>> r255131 - create known issues.  This one is to be reverted if all 3 types
>> show up properly.
>>
>> On Wed, Dec 9, 2015 at 9:41 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
>>
>>> It is a small change.
>>>
>>> I almost have all the trial tests ready, so I'll just commit both
>>> changes at the same time (the flip on, and the trial balloon issues).
>>>
>>> If all goes well and the three types of issue show up, then the last of
>>> the two will get reverted (the one with the failures).
>>>
>>> If none (or only some) of the issues show up, they'll both get reverted.
>>>
>>> -Todd
>>>
>>> On Wed, Dec 9, 2015 at 9:39 AM, Pavel Labath <labath at google.com> wrote:
>>>
>>>> If it's not too much work, I think the extra bit of noise will not be
>>>> a problem. But I don't think it is really necessary either.
>>>>
>>>> I assume the actual flip will be a small change that we can back out
>>>> easily if we notice troubles... After a sufficient grace period we can
>>>> remove the old formatter altogether and hopefully simplify the code
>>>> somewhat.
>>>>
>>>> pl
>>>>
>>>> On 9 December 2015 at 17:08, Todd Fiala <todd.fiala at gmail.com> wrote:
>>>> > Here's what I can do.
>>>> >
>>>> > Put in the change (setting the default to use the new format).
>>>> >
>>>> > Separately, put in a trial balloon commit with one failing test, one
>>>> > exceptional exit test, and one timeout test, and watch the ubuntu
>>>> 14.04
>>>> > buildbot catch it and fail.  Then reverse this out.  That should show
>>>> beyond
>>>> > a reasonable doubt whether the buildbot catches new failures and
>>>> errors.  (I
>>>> > think this is a noisy way to accomplish this, but it certainly would
>>>> > validate if its working).
>>>> >
>>>> > -Todd
>>>> >
>>>> > On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala <todd.fiala at gmail.com>
>>>> wrote:
>>>> >>
>>>> >> Specifically, the markers for issue details are:
>>>> >>
>>>> >> FAIL
>>>> >> ERROR
>>>> >> UNEXPECTED SUCCESS
>>>> >> TIMEOUT
>>>> >>
>>>> >> (These are the fourth field in the array entries (lines 275 - 290) of
>>>> >> packages/Python/lldbsuite/test/basic_results_formatter.py).
>>>> >>
>>>> >> -Todd
>>>> >>
>>>> >> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala <todd.fiala at gmail.com>
>>>> wrote:
>>>> >>>
>>>> >>> That's a good point, Tamas.
>>>> >>>
>>>> >>> I use (so I claim) the same all upper-case markers for the test
>>>> result
>>>> >>> details.  Including, not using XPASS but rather UNEXPECTED SUCCESS
>>>> for
>>>> >>> unexpected successes.  (The former would trigger the lit script
>>>> IIRC to
>>>> >>> parse that as a failing-style result).
>>>> >>>
>>>> >>> The intent is this is a no-op on the test runner.
>>>> >>>
>>>> >>> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer <
>>>> tberghammer at google.com>
>>>> >>> wrote:
>>>> >>>>
>>>> >>>> +Ying Chen
>>>> >>>>
>>>> >>>> Ying, what do we have to do on the build bot side to support a
>>>> change in
>>>> >>>> the default test result summary formatter?
>>>> >>>>
>>>> >>>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev
>>>> >>>> <lldb-dev at lists.llvm.org> wrote:
>>>> >>>>>
>>>> >>>>> Hi all,
>>>> >>>>>
>>>> >>>>> Per a previous thread on this, I've made all the changes I
>>>> intended to
>>>> >>>>> make last night to get the intended replacement of test run
>>>> results meet or
>>>> >>>>> exceed current requirements.
>>>> >>>>>
>>>> >>>>> I'd like to switch over to that by default.  I'm depending on the
>>>> test
>>>> >>>>> event system to be able to handle test method reruns in test
>>>> results
>>>> >>>>> accounting.
>>>> >>>>>
>>>> >>>>> The primary thing missing before was that timeouts were not routed
>>>> >>>>> through the test events system, nor were exception process exits
>>>> (i.e. test
>>>> >>>>> inferiors exiting with a signal on POSIX systems).  Those were
>>>> added last
>>>> >>>>> night so that test events are generated for those, and the
>>>> >>>>> BasicResultsFormatter presents that information properly.
>>>> >>>>>
>>>> >>>>> I will switch it over to being the default output in a bit here.
>>>> >>>>> Please let me know if you have any concerns once I flip it on by
>>>> default.
>>>> >>>>>
>>>> >>>>> Thanks!
>>>> >>>>> --
>>>> >>>>> -Todd
>>>> >>>>> _______________________________________________
>>>> >>>>> lldb-dev mailing list
>>>> >>>>> lldb-dev at lists.llvm.org
>>>> >>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> -Todd
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> -Todd
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > -Todd
>>>>
>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
>>
>>
>> --
>> -Todd
>>
>
>
>
> --
> -Todd
>

-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151209/7988ff31/attachment-0001.html>