[lldb-dev] New test summary results formatter

Wed Dec 2 11:32:43 PST 2015

Oh yea, I made up that decorator idea because I didn't know all the
formatters were derived from a common base.  But your idea is better if
everything is derived from a common base.  To be honest you could even just
generate an error if there are two ResultsFormatter derived classes in the
same module.  We should be encouraging more smaller files with single
responsibility.  One of the things I plan to do as part of some cleanup in
a week or two is to split up dotest, dosep, and lldbtest.py into a couple
different files by breaking out things like TestBase, etc into separate
files.  So that it's easier to keep a mental map of where different code is.

On Wed, Dec 2, 2015 at 11:26 AM Todd Fiala <todd.fiala at gmail.com> wrote:

> On Wed, Dec 2, 2015 at 11:20 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> Yeah I'd be good with that.  I can change that as well.
>>
>> -Todd
>>
>> On Wed, Dec 2, 2015 at 11:10 AM, Zachary Turner <zturner at google.com>
>> wrote:
>>
>>> Also another stylistic suggestion.  I've been thinking about how to more
>>> logically organize all the source files now that we have a package.  So it
>>> makes sense conceptually to group all of the different result formatters
>>> under a subpackage called formatters.  So right now you've got
>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter but it
>>> might make sense for this to be
>>> lldbsuite.test.formatters.basic.BasicResultsFormatter.  If you do things
>>> this way, it can actually result in a substantially shorter command line,
>>> because the --results-formatter option can use lldbsuite.test.formatters as
>>> a starting point.  So you could instead write:
>>>
>>> test/dotest.py --results-formatter basic
>>>
>>> dotest then looks for a `basic.py` module in the
>>> `lldbsuite.test.formatters` package, looks for a class inside with a
>>> @result_formatter decorator, and instantiates that.
>>>
>>> This has the advantage of making the command line shorter *and* a more
>>> logical source file organization.
>>>
>>
> The other thing that could allow me to do is possibly short-circuit the
> results formatter specifier so that, if just the module is specified, and
> if the module only has one ResultsFormatter-derived class, I can probably
> rig up code that figures out the right results formatter, shortening the
> required discriminator to something even shorter (i.e. module.classname
> becomes just module.)
>
>
>>
>>> On Wed, Dec 2, 2015 at 11:04 AM Zachary Turner <zturner at google.com>
>>> wrote:
>>>
>>>> Can --results-file=stdout be the default so that we don't have to
>>>> specify that?
>>>>
>>>> On Wed, Dec 2, 2015 at 11:02 AM Todd Fiala via lldb-dev <
>>>> lldb-dev at lists.llvm.org> wrote:
>>>>
>>>>> Also, all the text in the summary is fixed-width lined up nicely,
>>>>> which may not show in the commit message description if you're using a
>>>>> variable-width font.  On a terminal it looks nice.
>>>>>
>>>>> On Wed, Dec 2, 2015 at 11:01 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Dec 2, 2015 at 10:57 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I just put up an optional test results formatter that is a prototype
>>>>>>> of what we may move towards for our default test summary results.  It went
>>>>>>> in here:
>>>>>>>
>>>>>>> r254530
>>>>>>>
>>>>>>> and you can try it out with something like:
>>>>>>>
>>>>>>> time test/dotest.py --executable `pwd`/build/Debug/lldb
>>>>>>> --results-formatter
>>>>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file
>>>>>>> st
>>>>>>> out
>>>>>>>
>>>>>>>
>>>>>> I cut and paste my line, but more than likely for most people you'd
>>>>>> just want this:
>>>>>>
>>>>>> test/dotest.py --results-formatter
>>>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file
>>>>>> stdout
>>>>>>
>>>>>> The other stuff was specific to my setup.  That line assumes you run
>>>>>> from the lldb source dir root.
>>>>>>
>>>>>>
>>>>>> Let me know if this satisfies the basic needs of counts and whatnot.
>>>>>>> It counts test method runs rather than all the oddball "file, class, etc."
>>>>>>> counts we had before.
>>>>>>>
>>>>>>> It prints out the Details section when there are details, and keeps
>>>>>>> it nice and clean when there are none.
>>>>>>>
>>>>>>> It also mentions a bit about test reruns up top, but that won't come
>>>>>>> into play until I get the multi-test-pass, single-worker/low-load mechanism
>>>>>>> in place, which will depend on newer rerun count awareness support.
>>>>>>>
>>>>>>> The change also cleans up places where the test event framework was
>>>>>>> using string codes and replaces them with symbolic constants.
>>>>>>>
>>>>>>> Let me know what you think.  I can tweak it as needed to address
>>>>>>> testbot and other needs.  Once it looks reasonable, I'd like to move over
>>>>>>> to using it by default in the parallel test runner rather than the legacy
>>>>>>> support.
>>>>>>>
>>>>>>> Thanks!
>>>>>>> --
>>>>>>> -Todd
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -Todd
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Todd
>>>>> _______________________________________________
>>>>> lldb-dev mailing list
>>>>> lldb-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>
>>>>
>>
>>
>> --
>> -Todd
>>
>
>
>
> --
> -Todd
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151202/bda70642/attachment.html>