[lldb-dev] BasicResultsFormatter - new test results summary

Wed Dec 9 10:28:33 PST 2015

Verification tests parked (i.e. disabled) here:
r255134

I decided to leave them in the repo so it is faster/easier to do this in
the future.

-Todd

On Wed, Dec 9, 2015 at 10:26 AM, Todd Fiala <todd.fiala at gmail.com> wrote:

> The reports look good at the test level:
>
>
> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9294
>
> I'd say the buildbot reflection script missed the ERROR, so that is
> something maybe Ying can look at (the summary line in the build run), but
> that is unrelated AFAICT.
>
> I'm going to move aside the failures.
>
> On Wed, Dec 9, 2015 at 10:13 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> I am going to stop the current build on that builder.  There was one
>> change in it, and it will be another 20 minutes before it completes.  I
>> don't want the repo in a known broken state that long.
>>
>> On Wed, Dec 9, 2015 at 10:07 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
>>
>>> I forced a build on the ubuntu 14.04 cmake builder.  The build _after_
>>> 9292 will contain the two changes (and we will expect failures on it).
>>>
>>> On Wed, Dec 9, 2015 at 10:05 AM, Todd Fiala <todd.fiala at gmail.com>
>>> wrote:
>>>
>>>> These went in as:
>>>>
>>>> r255130 - turn it on by default
>>>> r255131 - create known issues.  This one is to be reverted if all 3
>>>> types show up properly.
>>>>
>>>> On Wed, Dec 9, 2015 at 9:41 AM, Todd Fiala <todd.fiala at gmail.com>
>>>> wrote:
>>>>
>>>>> It is a small change.
>>>>>
>>>>> I almost have all the trial tests ready, so I'll just commit both
>>>>> changes at the same time (the flip on, and the trial balloon issues).
>>>>>
>>>>> If all goes well and the three types of issue show up, then the last
>>>>> of the two will get reverted (the one with the failures).
>>>>>
>>>>> If none (or only some) of the issues show up, they'll both get
>>>>> reverted.
>>>>>
>>>>> -Todd
>>>>>
>>>>> On Wed, Dec 9, 2015 at 9:39 AM, Pavel Labath <labath at google.com>
>>>>> wrote:
>>>>>
>>>>>> If it's not too much work, I think the extra bit of noise will not be
>>>>>> a problem. But I don't think it is really necessary either.
>>>>>>
>>>>>> I assume the actual flip will be a small change that we can back out
>>>>>> easily if we notice troubles... After a sufficient grace period we can
>>>>>> remove the old formatter altogether and hopefully simplify the code
>>>>>> somewhat.
>>>>>>
>>>>>> pl
>>>>>>
>>>>>> On 9 December 2015 at 17:08, Todd Fiala <todd.fiala at gmail.com> wrote:
>>>>>> > Here's what I can do.
>>>>>> >
>>>>>> > Put in the change (setting the default to use the new format).
>>>>>> >
>>>>>> > Separately, put in a trial balloon commit with one failing test, one
>>>>>> > exceptional exit test, and one timeout test, and watch the ubuntu
>>>>>> 14.04
>>>>>> > buildbot catch it and fail.  Then reverse this out.  That should
>>>>>> show beyond
>>>>>> > a reasonable doubt whether the buildbot catches new failures and
>>>>>> errors.  (I
>>>>>> > think this is a noisy way to accomplish this, but it certainly would
>>>>>> > validate if its working).
>>>>>> >
>>>>>> > -Todd
>>>>>> >
>>>>>> > On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> Specifically, the markers for issue details are:
>>>>>> >>
>>>>>> >> FAIL
>>>>>> >> ERROR
>>>>>> >> UNEXPECTED SUCCESS
>>>>>> >> TIMEOUT
>>>>>> >>
>>>>>> >> (These are the fourth field in the array entries (lines 275 - 290)
>>>>>> of
>>>>>> >> packages/Python/lldbsuite/test/basic_results_formatter.py).
>>>>>> >>
>>>>>> >> -Todd
>>>>>> >>
>>>>>> >> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> That's a good point, Tamas.
>>>>>> >>>
>>>>>> >>> I use (so I claim) the same all upper-case markers for the test
>>>>>> result
>>>>>> >>> details.  Including, not using XPASS but rather UNEXPECTED
>>>>>> SUCCESS for
>>>>>> >>> unexpected successes.  (The former would trigger the lit script
>>>>>> IIRC to
>>>>>> >>> parse that as a failing-style result).
>>>>>> >>>
>>>>>> >>> The intent is this is a no-op on the test runner.
>>>>>> >>>
>>>>>> >>> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer <
>>>>>> tberghammer at google.com>
>>>>>> >>> wrote:
>>>>>> >>>>
>>>>>> >>>> +Ying Chen
>>>>>> >>>>
>>>>>> >>>> Ying, what do we have to do on the build bot side to support a
>>>>>> change in
>>>>>> >>>> the default test result summary formatter?
>>>>>> >>>>
>>>>>> >>>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev
>>>>>> >>>> <lldb-dev at lists.llvm.org> wrote:
>>>>>> >>>>>
>>>>>> >>>>> Hi all,
>>>>>> >>>>>
>>>>>> >>>>> Per a previous thread on this, I've made all the changes I
>>>>>> intended to
>>>>>> >>>>> make last night to get the intended replacement of test run
>>>>>> results meet or
>>>>>> >>>>> exceed current requirements.
>>>>>> >>>>>
>>>>>> >>>>> I'd like to switch over to that by default.  I'm depending on
>>>>>> the test
>>>>>> >>>>> event system to be able to handle test method reruns in test
>>>>>> results
>>>>>> >>>>> accounting.
>>>>>> >>>>>
>>>>>> >>>>> The primary thing missing before was that timeouts were not
>>>>>> routed
>>>>>> >>>>> through the test events system, nor were exception process
>>>>>> exits (i.e. test
>>>>>> >>>>> inferiors exiting with a signal on POSIX systems).  Those were
>>>>>> added last
>>>>>> >>>>> night so that test events are generated for those, and the
>>>>>> >>>>> BasicResultsFormatter presents that information properly.
>>>>>> >>>>>
>>>>>> >>>>> I will switch it over to being the default output in a bit here.
>>>>>> >>>>> Please let me know if you have any concerns once I flip it on
>>>>>> by default.
>>>>>> >>>>>
>>>>>> >>>>> Thanks!
>>>>>> >>>>> --
>>>>>> >>>>> -Todd
>>>>>> >>>>> _______________________________________________
>>>>>> >>>>> lldb-dev mailing list
>>>>>> >>>>> lldb-dev at lists.llvm.org
>>>>>> >>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> --
>>>>>> >>> -Todd
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> --
>>>>>> >> -Todd
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > -Todd
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Todd
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> -Todd
>>>>
>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
>>
>>
>> --
>> -Todd
>>
>
>
>
> --
> -Todd
>

-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151209/a5dfe70d/attachment-0001.html>