[lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

Hi all,

r254890 moves the test summary counts to the end.  It also greatly cleans
up the issue detail line to be:

ISSUE_TYPE: test_method_name (test relative path)

I put a sample output in the revision comment.  I think it looks much
cleaner with the tweaks we discussed, and I really like the look of the
counts at the very end.

I'll work on getting the timeouts and exceptional exits fed into the
output, in addition to moving those two straggling counts from above the
issue details and moved into the summary counts.

After that, we can evaluate if we want to switch over to this as the
default output.  Then I can get the low-load, single worker test pass to be
added to the end of the test run.

-Todd



On Fri, Dec 4, 2015 at 5:33 PM, Todd Fiala <todd.fiala at gmail.com> wrote:

> One thing I excluded from the newer test results detail info is the
> architecture.  I personally haven't ever needed that.  I'd be happy to
> leave that out until we find someone who really needs it, just to keep it
> shorter.
>
> On Thu, Dec 3, 2015 at 5:14 PM, Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> That seems reasonable. I'll work that in.
>>
>> -Todd
>>
>> On Dec 3, 2015, at 4:55 PM, Zachary Turner <zturner at google.com> wrote:
>>
>> It would also be nice if the summary statistics were printed after the
>> list of failing / errored tests.  The reason is that it involves a fixed
>> number of lines to print the table, but the list of failures and errors is
>> a variable number of lines which could potentially be very long and push
>> the statistics off the screen.
>>
>> On Thu, Dec 3, 2015 at 10:08 AM Zachary Turner <zturner at google.com>
>> wrote:
>>
>>> Ahh I read further and see this was already mentioned by Pavel.
>>>
>>> On Thu, Dec 3, 2015 at 10:06 AM Zachary Turner <zturner at google.com>
>>> wrote:
>>>
>>>> On Wed, Dec 2, 2015 at 10:20 PM Todd Fiala <todd.fiala at gmail.com>
>>>> wrote:
>>>>
>>>>> On Wed, Dec 2, 2015 at 9:48 PM, Zachary Turner <zturner at google.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Dec 2, 2015 at 9:44 PM Todd Fiala <todd.fiala at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> and the classname could be dropped (there's only one class per file
>>>>>>>> anyway, so the classname is just wasted space)
>>>>>>>>
>>>>>>>
>>>>>>> Part of the reason I included that is I've hit several times where
>>>>>>> copy and paste errors lead to the same class name, method name or even file
>>>>>>> name being used for a test.  I think, though, that most of those are
>>>>>>> addressed by having the path (relative is fine) to the python test file.  I
>>>>>>> think we can probably get by with classname.methodname (relative test
>>>>>>> path).  (From your other email, I think you nuke the classname and keep the
>>>>>>> module name, but I'd probably do the reverse, keeping the class name and
>>>>>>> getting rid of the module name since it can be derived from the filename).
>>>>>>>
>>>>>> I don't think the filename can be the same anymore, as things will
>>>>>> break if two filenames are the same.
>>>>>>
>>>>>
>>>>> Maybe, but that wasn't my experience as of fairly recently.  When
>>>>> tracking failures sometime within the last month, I tracked something down
>>>>> in a downstream branch with two same-named files that (with the legacy
>>>>> output) made it hard to track down what was actually failing given the
>>>>> limited info of the legacy test summary output.  Maybe that has changed
>>>>> since then, but I'm not aware of anything that would have prohibited that.
>>>>>
>>>> Well I only said "things" will break, not everything will break.  Most
>>>> likely you just didn't notice the problem or it didn't present itself in
>>>> your scenario.  There are definitely bugs surrounding multiple files with
>>>> the same name, because of some places where we use a dictionary keyed on
>>>> filename.
>>>>
>>>>
>>>>>
>
>
> --
> -Todd
>



-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151206/857b9a86/attachment.html>