[lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

Here's what I can do.

Put in the change (setting the default to use the new format).

Separately, put in a trial balloon commit with one failing test, one
exceptional exit test, and one timeout test, and watch the ubuntu 14.04
buildbot catch it and fail.  Then reverse this out.  That should show
beyond a reasonable doubt whether the buildbot catches new failures and
errors.  (I think this is a noisy way to accomplish this, but it certainly
would validate if its working).

-Todd

On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala <todd.fiala at gmail.com> wrote:

> Specifically, the markers for issue details are:
>
> FAIL
> ERROR
> UNEXPECTED SUCCESS
> TIMEOUT
>
> (These are the fourth field in the array entries (lines 275 - 290) of
> packages/Python/lldbsuite/test/basic_results_formatter.py).
>
> -Todd
>
> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> That's a good point, Tamas.
>>
>> I use (so I claim) the same all upper-case markers for the test result
>> details.  Including, not using XPASS but rather UNEXPECTED SUCCESS for
>> unexpected successes.  (The former would trigger the lit script IIRC to
>> parse that as a failing-style result).
>>
>> The intent is this is a no-op on the test runner.
>>
>> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer <tberghammer at google.com>
>> wrote:
>>
>>> +Ying Chen <chying at google.com>
>>>
>>> Ying, what do we have to do on the build bot side to support a change in
>>> the default test result summary formatter?
>>>
>>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev <
>>> lldb-dev at lists.llvm.org> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Per a previous thread on this, I've made all the changes I intended to
>>>> make last night to get the intended replacement of test run results meet or
>>>> exceed current requirements.
>>>>
>>>> I'd like to switch over to that by default.  I'm depending on the test
>>>> event system to be able to handle test method reruns in test results
>>>> accounting.
>>>>
>>>> The primary thing missing before was that timeouts were not routed
>>>> through the test events system, nor were exception process exits (i.e. test
>>>> inferiors exiting with a signal on POSIX systems).  Those were added last
>>>> night so that test events are generated for those, and the
>>>> BasicResultsFormatter presents that information properly.
>>>>
>>>> I will switch it over to being the default output in a bit here.
>>>> Please let me know if you have any concerns once I flip it on by default.
>>>>
>>>> Thanks!
>>>> --
>>>> -Todd
>>>> _______________________________________________
>>>> lldb-dev mailing list
>>>> lldb-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>
>>>
>>
>>
>> --
>> -Todd
>>
>
>
>
> --
> -Todd
>



-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151209/cc1189ec/attachment-0001.html>