[lldb-dev] BasicResultsFormatter - new test results summary

Thu Dec 10 12:54:02 PST 2015

Hi Tamas,

On Thu, Dec 10, 2015 at 2:52 AM, Tamas Berghammer <tberghammer at google.com>
wrote:

> HI Todd,
>
> You changed the way the test failure list is printed in a way that now we
> only print the name of the test function failing with the name of the test
> file in parenthesis. Can we add back the name of the test class to this
> list?
>

Sure.  I originally planned to have that in there but there was some
discussion about it being too much info.  I'm happy to add that back.

>
> There are 2 reason I am asking for it:
> * To run only a specific test we have to specify the "-f" option to
> dotest.py and it takes the fully qualified function name as an argument.
> Before your change it was displayed in the test output (in a bit
> uncomfortable way) but after your change we have to open the test file and
> copy the class name from there to run only a single test suit.
> * With the new output format the output of the buildbot only displays the
> list of the failing test function names what isn't too specific in a lot of
> case (e.g. we have several test method called test_dwarf). This point is
> less important as the file name can be added to the output from the
> buildbot perspective.
>

Okay, not sure exactly what is doing that printing, but the file name is
available from the details of the summary.  Sounds like it is a text
parsing change needed based on the changes I made to the format.

>
> Thanks,
>

Sure thing!

-Todd

> Tamas
>
> On Wed, Dec 9, 2015 at 7:57 PM Ying Chen <chying at google.com> wrote:
>
>> I submitted this patch to include "ERROR" lines in buildbot step results.
>> http://reviews.llvm.org/rL255145
>>
>> Error results will be displayed in step result like this after the patch,
>> "ERROR: 9 (SIGKILL) test_buildbot_catches_exceptional_exit_dwarf"
>>
>> Thanks,
>> Ying
>>
>> On Wed, Dec 9, 2015 at 10:45 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
>>
>>> Great, thanks Tamas!
>>>
>>> I left the default turned on, and just essentially removed the issues by
>>> parking them as .py.parked files.  That way we can flip them on in the
>>> future if we want to verify a testbot's detection of these.
>>>
>>> I will be going back to the xUnit Results formatter and making sure it
>>> maps timeouts and exceptional errors to the xUnit error type with details.
>>>
>>> On Wed, Dec 9, 2015 at 10:30 AM, Tamas Berghammer <
>>> tberghammer at google.com> wrote:
>>>
>>>> Thank you for making the experiment. It looks reasonable. For the ERROR
>>>> the buildbot detected it and it will fail the build but it isn't listed in
>>>> the list of failing tests what should be fixed. After this experiment I
>>>> think it is fine to change the default output formatter from our side.
>>>>
>>>> Tamas
>>>>
>>>> On Wed, Dec 9, 2015 at 6:26 PM Todd Fiala <todd.fiala at gmail.com> wrote:
>>>>
>>>>> The reports look good at the test level:
>>>>>
>>>>>
>>>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9294
>>>>>
>>>>> I'd say the buildbot reflection script missed the ERROR, so that is
>>>>> something maybe Ying can look at (the summary line in the build run), but
>>>>> that is unrelated AFAICT.
>>>>>
>>>>> I'm going to move aside the failures.
>>>>>
>>>>> On Wed, Dec 9, 2015 at 10:13 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I am going to stop the current build on that builder.  There was one
>>>>>> change in it, and it will be another 20 minutes before it completes.  I
>>>>>> don't want the repo in a known broken state that long.
>>>>>>
>>>>>> On Wed, Dec 9, 2015 at 10:07 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I forced a build on the ubuntu 14.04 cmake builder.  The build
>>>>>>> _after_ 9292 will contain the two changes (and we will expect failures on
>>>>>>> it).
>>>>>>>
>>>>>>> On Wed, Dec 9, 2015 at 10:05 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> These went in as:
>>>>>>>>
>>>>>>>> r255130 - turn it on by default
>>>>>>>> r255131 - create known issues.  This one is to be reverted if all 3
>>>>>>>> types show up properly.
>>>>>>>>
>>>>>>>> On Wed, Dec 9, 2015 at 9:41 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> It is a small change.
>>>>>>>>>
>>>>>>>>> I almost have all the trial tests ready, so I'll just commit both
>>>>>>>>> changes at the same time (the flip on, and the trial balloon issues).
>>>>>>>>>
>>>>>>>>> If all goes well and the three types of issue show up, then the
>>>>>>>>> last of the two will get reverted (the one with the failures).
>>>>>>>>>
>>>>>>>>> If none (or only some) of the issues show up, they'll both get
>>>>>>>>> reverted.
>>>>>>>>>
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>> On Wed, Dec 9, 2015 at 9:39 AM, Pavel Labath <labath at google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> If it's not too much work, I think the extra bit of noise will
>>>>>>>>>> not be
>>>>>>>>>> a problem. But I don't think it is really necessary either.
>>>>>>>>>>
>>>>>>>>>> I assume the actual flip will be a small change that we can back
>>>>>>>>>> out
>>>>>>>>>> easily if we notice troubles... After a sufficient grace period
>>>>>>>>>> we can
>>>>>>>>>> remove the old formatter altogether and hopefully simplify the
>>>>>>>>>> code
>>>>>>>>>> somewhat.
>>>>>>>>>>
>>>>>>>>>> pl
>>>>>>>>>>
>>>>>>>>>> On 9 December 2015 at 17:08, Todd Fiala <todd.fiala at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>> > Here's what I can do.
>>>>>>>>>> >
>>>>>>>>>> > Put in the change (setting the default to use the new format).
>>>>>>>>>> >
>>>>>>>>>> > Separately, put in a trial balloon commit with one failing
>>>>>>>>>> test, one
>>>>>>>>>> > exceptional exit test, and one timeout test, and watch the
>>>>>>>>>> ubuntu 14.04
>>>>>>>>>> > buildbot catch it and fail.  Then reverse this out.  That
>>>>>>>>>> should show beyond
>>>>>>>>>> > a reasonable doubt whether the buildbot catches new failures
>>>>>>>>>> and errors.  (I
>>>>>>>>>> > think this is a noisy way to accomplish this, but it certainly
>>>>>>>>>> would
>>>>>>>>>> > validate if its working).
>>>>>>>>>> >
>>>>>>>>>> > -Todd
>>>>>>>>>> >
>>>>>>>>>> > On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala <
>>>>>>>>>> todd.fiala at gmail.com> wrote:
>>>>>>>>>> >>
>>>>>>>>>> >> Specifically, the markers for issue details are:
>>>>>>>>>> >>
>>>>>>>>>> >> FAIL
>>>>>>>>>> >> ERROR
>>>>>>>>>> >> UNEXPECTED SUCCESS
>>>>>>>>>> >> TIMEOUT
>>>>>>>>>> >>
>>>>>>>>>> >> (These are the fourth field in the array entries (lines 275 -
>>>>>>>>>> 290) of
>>>>>>>>>> >> packages/Python/lldbsuite/test/basic_results_formatter.py).
>>>>>>>>>> >>
>>>>>>>>>> >> -Todd
>>>>>>>>>> >>
>>>>>>>>>> >> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala <
>>>>>>>>>> todd.fiala at gmail.com> wrote:
>>>>>>>>>> >>>
>>>>>>>>>> >>> That's a good point, Tamas.
>>>>>>>>>> >>>
>>>>>>>>>> >>> I use (so I claim) the same all upper-case markers for the
>>>>>>>>>> test result
>>>>>>>>>> >>> details.  Including, not using XPASS but rather UNEXPECTED
>>>>>>>>>> SUCCESS for
>>>>>>>>>> >>> unexpected successes.  (The former would trigger the lit
>>>>>>>>>> script IIRC to
>>>>>>>>>> >>> parse that as a failing-style result).
>>>>>>>>>> >>>
>>>>>>>>>> >>> The intent is this is a no-op on the test runner.
>>>>>>>>>> >>>
>>>>>>>>>> >>> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer <
>>>>>>>>>> tberghammer at google.com>
>>>>>>>>>> >>> wrote:
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> +Ying Chen
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> Ying, what do we have to do on the build bot side to support
>>>>>>>>>> a change in
>>>>>>>>>> >>>> the default test result summary formatter?
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev
>>>>>>>>>> >>>> <lldb-dev at lists.llvm.org> wrote:
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> Hi all,
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> Per a previous thread on this, I've made all the changes I
>>>>>>>>>> intended to
>>>>>>>>>> >>>>> make last night to get the intended replacement of test run
>>>>>>>>>> results meet or
>>>>>>>>>> >>>>> exceed current requirements.
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> I'd like to switch over to that by default.  I'm depending
>>>>>>>>>> on the test
>>>>>>>>>> >>>>> event system to be able to handle test method reruns in
>>>>>>>>>> test results
>>>>>>>>>> >>>>> accounting.
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> The primary thing missing before was that timeouts were not
>>>>>>>>>> routed
>>>>>>>>>> >>>>> through the test events system, nor were exception process
>>>>>>>>>> exits (i.e. test
>>>>>>>>>> >>>>> inferiors exiting with a signal on POSIX systems).  Those
>>>>>>>>>> were added last
>>>>>>>>>> >>>>> night so that test events are generated for those, and the
>>>>>>>>>> >>>>> BasicResultsFormatter presents that information properly.
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> I will switch it over to being the default output in a bit
>>>>>>>>>> here.
>>>>>>>>>> >>>>> Please let me know if you have any concerns once I flip it
>>>>>>>>>> on by default.
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> Thanks!
>>>>>>>>>> >>>>> --
>>>>>>>>>> >>>>> -Todd
>>>>>>>>>> >>>>> _______________________________________________
>>>>>>>>>> >>>>> lldb-dev mailing list
>>>>>>>>>> >>>>> lldb-dev at lists.llvm.org
>>>>>>>>>> >>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>>>> >>>
>>>>>>>>>> >>>
>>>>>>>>>> >>>
>>>>>>>>>> >>>
>>>>>>>>>> >>> --
>>>>>>>>>> >>> -Todd
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >> --
>>>>>>>>>> >> -Todd
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > --
>>>>>>>>>> > -Todd
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> -Todd
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -Todd
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Todd
>>>>>
>>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
>>

-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151210/03b19505/attachment.html>