[lldb-dev] New test summary results formatter

Wed Dec 2 11:40:37 PST 2015

When I run this under Python 3 I get "A bytes object is used like a string"
on Line 1033 of test_results.py.  I'm going to dig into it a little bit,
but maybe you know off the top of your head the right way to fix it.

On Wed, Dec 2, 2015 at 11:32 AM Zachary Turner <zturner at google.com> wrote:

> Oh yea, I made up that decorator idea because I didn't know all the
> formatters were derived from a common base.  But your idea is better if
> everything is derived from a common base.  To be honest you could even just
> generate an error if there are two ResultsFormatter derived classes in the
> same module.  We should be encouraging more smaller files with single
> responsibility.  One of the things I plan to do as part of some cleanup in
> a week or two is to split up dotest, dosep, and lldbtest.py into a couple
> different files by breaking out things like TestBase, etc into separate
> files.  So that it's easier to keep a mental map of where different code is.
>
> On Wed, Dec 2, 2015 at 11:26 AM Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> On Wed, Dec 2, 2015 at 11:20 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
>>
>>> Yeah I'd be good with that.  I can change that as well.
>>>
>>> -Todd
>>>
>>> On Wed, Dec 2, 2015 at 11:10 AM, Zachary Turner <zturner at google.com>
>>> wrote:
>>>
>>>> Also another stylistic suggestion.  I've been thinking about how to
>>>> more logically organize all the source files now that we have a package.
>>>> So it makes sense conceptually to group all of the different result
>>>> formatters under a subpackage called formatters.  So right now you've got
>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter but it
>>>> might make sense for this to be
>>>> lldbsuite.test.formatters.basic.BasicResultsFormatter.  If you do things
>>>> this way, it can actually result in a substantially shorter command line,
>>>> because the --results-formatter option can use lldbsuite.test.formatters as
>>>> a starting point.  So you could instead write:
>>>>
>>>> test/dotest.py --results-formatter basic
>>>>
>>>> dotest then looks for a `basic.py` module in the
>>>> `lldbsuite.test.formatters` package, looks for a class inside with a
>>>> @result_formatter decorator, and instantiates that.
>>>>
>>>> This has the advantage of making the command line shorter *and* a more
>>>> logical source file organization.
>>>>
>>>
>> The other thing that could allow me to do is possibly short-circuit the
>> results formatter specifier so that, if just the module is specified, and
>> if the module only has one ResultsFormatter-derived class, I can probably
>> rig up code that figures out the right results formatter, shortening the
>> required discriminator to something even shorter (i.e. module.classname
>> becomes just module.)
>>
>>
>>>
>>>> On Wed, Dec 2, 2015 at 11:04 AM Zachary Turner <zturner at google.com>
>>>> wrote:
>>>>
>>>>> Can --results-file=stdout be the default so that we don't have to
>>>>> specify that?
>>>>>
>>>>> On Wed, Dec 2, 2015 at 11:02 AM Todd Fiala via lldb-dev <
>>>>> lldb-dev at lists.llvm.org> wrote:
>>>>>
>>>>>> Also, all the text in the summary is fixed-width lined up nicely,
>>>>>> which may not show in the commit message description if you're using a
>>>>>> variable-width font.  On a terminal it looks nice.
>>>>>>
>>>>>> On Wed, Dec 2, 2015 at 11:01 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Dec 2, 2015 at 10:57 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I just put up an optional test results formatter that is a
>>>>>>>> prototype of what we may move towards for our default test summary
>>>>>>>> results.  It went in here:
>>>>>>>>
>>>>>>>> r254530
>>>>>>>>
>>>>>>>> and you can try it out with something like:
>>>>>>>>
>>>>>>>> time test/dotest.py --executable `pwd`/build/Debug/lldb
>>>>>>>> --results-formatter
>>>>>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file
>>>>>>>> st
>>>>>>>> out
>>>>>>>>
>>>>>>>>
>>>>>>> I cut and paste my line, but more than likely for most people you'd
>>>>>>> just want this:
>>>>>>>
>>>>>>> test/dotest.py --results-formatter
>>>>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file
>>>>>>> stdout
>>>>>>>
>>>>>>> The other stuff was specific to my setup.  That line assumes you run
>>>>>>> from the lldb source dir root.
>>>>>>>
>>>>>>>
>>>>>>> Let me know if this satisfies the basic needs of counts and
>>>>>>>> whatnot.  It counts test method runs rather than all the oddball "file,
>>>>>>>> class, etc." counts we had before.
>>>>>>>>
>>>>>>>> It prints out the Details section when there are details, and keeps
>>>>>>>> it nice and clean when there are none.
>>>>>>>>
>>>>>>>> It also mentions a bit about test reruns up top, but that won't
>>>>>>>> come into play until I get the multi-test-pass, single-worker/low-load
>>>>>>>> mechanism in place, which will depend on newer rerun count awareness
>>>>>>>> support.
>>>>>>>>
>>>>>>>> The change also cleans up places where the test event framework was
>>>>>>>> using string codes and replaces them with symbolic constants.
>>>>>>>>
>>>>>>>> Let me know what you think.  I can tweak it as needed to address
>>>>>>>> testbot and other needs.  Once it looks reasonable, I'd like to move over
>>>>>>>> to using it by default in the parallel test runner rather than the legacy
>>>>>>>> support.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> --
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> -Todd
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -Todd
>>>>>> _______________________________________________
>>>>>> lldb-dev mailing list
>>>>>> lldb-dev at lists.llvm.org
>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>
>>>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
>>
>>
>> --
>> -Todd
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151202/0347bae6/attachment-0001.html>