[lldb-dev] BasicResultsFormatter - new test results summary

[Pavel Labath via lldb-dev]

If it's not too much work, I think the extra bit of noise will not be
a problem. But I don't think it is really necessary either.

I assume the actual flip will be a small change that we can back out
easily if we notice troubles... After a sufficient grace period we can
remove the old formatter altogether and hopefully simplify the code
somewhat.

pl

On 9 December 2015 at 17:08, Todd Fiala <todd.fiala at gmail.com> wrote:
> Here's what I can do.
>
> Put in the change (setting the default to use the new format).
>
> Separately, put in a trial balloon commit with one failing test, one
> exceptional exit test, and one timeout test, and watch the ubuntu 14.04
> buildbot catch it and fail.  Then reverse this out.  That should show beyond
> a reasonable doubt whether the buildbot catches new failures and errors.  (I
> think this is a noisy way to accomplish this, but it certainly would
> validate if its working).
>
> -Todd
>
> On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
>>
>> Specifically, the markers for issue details are:
>>
>> FAIL
>> ERROR
>> UNEXPECTED SUCCESS
>> TIMEOUT
>>
>> (These are the fourth field in the array entries (lines 275 - 290) of
>> packages/Python/lldbsuite/test/basic_results_formatter.py).
>>
>> -Todd
>>
>> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
>>>
>>> That's a good point, Tamas.
>>>
>>> I use (so I claim) the same all upper-case markers for the test result
>>> details.  Including, not using XPASS but rather UNEXPECTED SUCCESS for
>>> unexpected successes.  (The former would trigger the lit script IIRC to
>>> parse that as a failing-style result).
>>>
>>> The intent is this is a no-op on the test runner.
>>>
>>> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer <tberghammer at google.com>
>>> wrote:
>>>>
>>>> +Ying Chen
>>>>
>>>> Ying, what do we have to do on the build bot side to support a change in
>>>> the default test result summary formatter?
>>>>
>>>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev
>>>> <lldb-dev at lists.llvm.org> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> Per a previous thread on this, I've made all the changes I intended to
>>>>> make last night to get the intended replacement of test run results meet or
>>>>> exceed current requirements.
>>>>>
>>>>> I'd like to switch over to that by default.  I'm depending on the test
>>>>> event system to be able to handle test method reruns in test results
>>>>> accounting.
>>>>>
>>>>> The primary thing missing before was that timeouts were not routed
>>>>> through the test events system, nor were exception process exits (i.e. test
>>>>> inferiors exiting with a signal on POSIX systems).  Those were added last
>>>>> night so that test events are generated for those, and the
>>>>> BasicResultsFormatter presents that information properly.
>>>>>
>>>>> I will switch it over to being the default output in a bit here.
>>>>> Please let me know if you have any concerns once I flip it on by default.
>>>>>
>>>>> Thanks!
>>>>> --
>>>>> -Todd
>>>>> _______________________________________________
>>>>> lldb-dev mailing list
>>>>> lldb-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>
>>>
>>>
>>>
>>> --
>>> -Todd
>>
>>
>>
>>
>> --
>> -Todd
>
>
>
>
> --
> -Todd