[lldb-dev] BasicResultsFormatter - new test results summary

Wed Dec 9 11:57:20 PST 2015

I submitted this patch to include "ERROR" lines in buildbot step results.
http://reviews.llvm.org/rL255145

Error results will be displayed in step result like this after the patch,
"ERROR: 9 (SIGKILL) test_buildbot_catches_exceptional_exit_dwarf"

Thanks,
Ying

On Wed, Dec 9, 2015 at 10:45 AM, Todd Fiala <todd.fiala at gmail.com> wrote:

> Great, thanks Tamas!
>
> I left the default turned on, and just essentially removed the issues by
> parking them as .py.parked files.  That way we can flip them on in the
> future if we want to verify a testbot's detection of these.
>
> I will be going back to the xUnit Results formatter and making sure it
> maps timeouts and exceptional errors to the xUnit error type with details.
>
> On Wed, Dec 9, 2015 at 10:30 AM, Tamas Berghammer <tberghammer at google.com>
> wrote:
>
>> Thank you for making the experiment. It looks reasonable. For the ERROR
>> the buildbot detected it and it will fail the build but it isn't listed in
>> the list of failing tests what should be fixed. After this experiment I
>> think it is fine to change the default output formatter from our side.
>>
>> Tamas
>>
>> On Wed, Dec 9, 2015 at 6:26 PM Todd Fiala <todd.fiala at gmail.com> wrote:
>>
>>> The reports look good at the test level:
>>>
>>>
>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9294
>>>
>>> I'd say the buildbot reflection script missed the ERROR, so that is
>>> something maybe Ying can look at (the summary line in the build run), but
>>> that is unrelated AFAICT.
>>>
>>> I'm going to move aside the failures.
>>>
>>> On Wed, Dec 9, 2015 at 10:13 AM, Todd Fiala <todd.fiala at gmail.com>
>>> wrote:
>>>
>>>> I am going to stop the current build on that builder.  There was one
>>>> change in it, and it will be another 20 minutes before it completes.  I
>>>> don't want the repo in a known broken state that long.
>>>>
>>>> On Wed, Dec 9, 2015 at 10:07 AM, Todd Fiala <todd.fiala at gmail.com>
>>>> wrote:
>>>>
>>>>> I forced a build on the ubuntu 14.04 cmake builder.  The build _after_
>>>>> 9292 will contain the two changes (and we will expect failures on it).
>>>>>
>>>>> On Wed, Dec 9, 2015 at 10:05 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> These went in as:
>>>>>>
>>>>>> r255130 - turn it on by default
>>>>>> r255131 - create known issues.  This one is to be reverted if all 3
>>>>>> types show up properly.
>>>>>>
>>>>>> On Wed, Dec 9, 2015 at 9:41 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> It is a small change.
>>>>>>>
>>>>>>> I almost have all the trial tests ready, so I'll just commit both
>>>>>>> changes at the same time (the flip on, and the trial balloon issues).
>>>>>>>
>>>>>>> If all goes well and the three types of issue show up, then the last
>>>>>>> of the two will get reverted (the one with the failures).
>>>>>>>
>>>>>>> If none (or only some) of the issues show up, they'll both get
>>>>>>> reverted.
>>>>>>>
>>>>>>> -Todd
>>>>>>>
>>>>>>> On Wed, Dec 9, 2015 at 9:39 AM, Pavel Labath <labath at google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> If it's not too much work, I think the extra bit of noise will not
>>>>>>>> be
>>>>>>>> a problem. But I don't think it is really necessary either.
>>>>>>>>
>>>>>>>> I assume the actual flip will be a small change that we can back out
>>>>>>>> easily if we notice troubles... After a sufficient grace period we
>>>>>>>> can
>>>>>>>> remove the old formatter altogether and hopefully simplify the code
>>>>>>>> somewhat.
>>>>>>>>
>>>>>>>> pl
>>>>>>>>
>>>>>>>> On 9 December 2015 at 17:08, Todd Fiala <todd.fiala at gmail.com>
>>>>>>>> wrote:
>>>>>>>> > Here's what I can do.
>>>>>>>> >
>>>>>>>> > Put in the change (setting the default to use the new format).
>>>>>>>> >
>>>>>>>> > Separately, put in a trial balloon commit with one failing test,
>>>>>>>> one
>>>>>>>> > exceptional exit test, and one timeout test, and watch the ubuntu
>>>>>>>> 14.04
>>>>>>>> > buildbot catch it and fail.  Then reverse this out.  That should
>>>>>>>> show beyond
>>>>>>>> > a reasonable doubt whether the buildbot catches new failures and
>>>>>>>> errors.  (I
>>>>>>>> > think this is a noisy way to accomplish this, but it certainly
>>>>>>>> would
>>>>>>>> > validate if its working).
>>>>>>>> >
>>>>>>>> > -Todd
>>>>>>>> >
>>>>>>>> > On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>>>>> wrote:
>>>>>>>> >>
>>>>>>>> >> Specifically, the markers for issue details are:
>>>>>>>> >>
>>>>>>>> >> FAIL
>>>>>>>> >> ERROR
>>>>>>>> >> UNEXPECTED SUCCESS
>>>>>>>> >> TIMEOUT
>>>>>>>> >>
>>>>>>>> >> (These are the fourth field in the array entries (lines 275 -
>>>>>>>> 290) of
>>>>>>>> >> packages/Python/lldbsuite/test/basic_results_formatter.py).
>>>>>>>> >>
>>>>>>>> >> -Todd
>>>>>>>> >>
>>>>>>>> >> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>>>>> wrote:
>>>>>>>> >>>
>>>>>>>> >>> That's a good point, Tamas.
>>>>>>>> >>>
>>>>>>>> >>> I use (so I claim) the same all upper-case markers for the test
>>>>>>>> result
>>>>>>>> >>> details.  Including, not using XPASS but rather UNEXPECTED
>>>>>>>> SUCCESS for
>>>>>>>> >>> unexpected successes.  (The former would trigger the lit script
>>>>>>>> IIRC to
>>>>>>>> >>> parse that as a failing-style result).
>>>>>>>> >>>
>>>>>>>> >>> The intent is this is a no-op on the test runner.
>>>>>>>> >>>
>>>>>>>> >>> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer <
>>>>>>>> tberghammer at google.com>
>>>>>>>> >>> wrote:
>>>>>>>> >>>>
>>>>>>>> >>>> +Ying Chen
>>>>>>>> >>>>
>>>>>>>> >>>> Ying, what do we have to do on the build bot side to support a
>>>>>>>> change in
>>>>>>>> >>>> the default test result summary formatter?
>>>>>>>> >>>>
>>>>>>>> >>>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev
>>>>>>>> >>>> <lldb-dev at lists.llvm.org> wrote:
>>>>>>>> >>>>>
>>>>>>>> >>>>> Hi all,
>>>>>>>> >>>>>
>>>>>>>> >>>>> Per a previous thread on this, I've made all the changes I
>>>>>>>> intended to
>>>>>>>> >>>>> make last night to get the intended replacement of test run
>>>>>>>> results meet or
>>>>>>>> >>>>> exceed current requirements.
>>>>>>>> >>>>>
>>>>>>>> >>>>> I'd like to switch over to that by default.  I'm depending on
>>>>>>>> the test
>>>>>>>> >>>>> event system to be able to handle test method reruns in test
>>>>>>>> results
>>>>>>>> >>>>> accounting.
>>>>>>>> >>>>>
>>>>>>>> >>>>> The primary thing missing before was that timeouts were not
>>>>>>>> routed
>>>>>>>> >>>>> through the test events system, nor were exception process
>>>>>>>> exits (i.e. test
>>>>>>>> >>>>> inferiors exiting with a signal on POSIX systems).  Those
>>>>>>>> were added last
>>>>>>>> >>>>> night so that test events are generated for those, and the
>>>>>>>> >>>>> BasicResultsFormatter presents that information properly.
>>>>>>>> >>>>>
>>>>>>>> >>>>> I will switch it over to being the default output in a bit
>>>>>>>> here.
>>>>>>>> >>>>> Please let me know if you have any concerns once I flip it on
>>>>>>>> by default.
>>>>>>>> >>>>>
>>>>>>>> >>>>> Thanks!
>>>>>>>> >>>>> --
>>>>>>>> >>>>> -Todd
>>>>>>>> >>>>> _______________________________________________
>>>>>>>> >>>>> lldb-dev mailing list
>>>>>>>> >>>>> lldb-dev at lists.llvm.org
>>>>>>>> >>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>> --
>>>>>>>> >>> -Todd
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> --
>>>>>>>> >> -Todd
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > -Todd
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> -Todd
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -Todd
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Todd
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> -Todd
>>>>
>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
>
>
> --
> -Todd
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151209/082f5842/attachment.html>