[lldb-dev] New test summary results formatter

Wed Dec 2 12:34:52 PST 2015

Is there any way to force the single process test runner to use this same
system?  I'm trying to debug the problem, but this codepath doesn't execute
in the single process test runner, and it executes in the child process in
the multiprocess test runner.  Basically I need the following callstack to
execute in the single process test runner:

Command invoked: C:\Python35\python_d.exe
D:\src\llvm\tools\lldb\test\dotest.py -q --arch=i686 --executable
D:/src/llvmbuild/ninja_py35/bin/lldb.exe -s
D:/src/llvmbuild/ninja_py35/lldb-test-traces -u CXXFLAGS -u CFLAGS
--enable-crash-dialog -C d:\src\llvmbuild\ninja_release\bin\clang.exe
--results-port 60794 --inferior -p TestIntegerTypesExpr.py
D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test --event-add-entries
worker_index=7:int
411 out of 412 test suites processed - TestIntegerTypesExpr.py
Traceback (most recent call last):
  File "D:\src\llvm\tools\lldb\test\dotest.py", line 7, in <module>
    lldbsuite.test.run_suite()
  File "D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\dotest.py",
line 1476, in run_suite
    setupTestResults()
  File "D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\dotest.py",
line 982, in setupTestResults
    results_formatter_object.handle_event(initialize_event)
  File
"D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\test_results.py",
line 1033, in handle_event
    "{}#{}".format(len(pickled_message), pickled_message))
TypeError: a bytes-like object is required, not 'str'

On Wed, Dec 2, 2015 at 11:40 AM Zachary Turner <zturner at google.com> wrote:

> When I run this under Python 3 I get "A bytes object is used like a
> string" on Line 1033 of test_results.py.  I'm going to dig into it a little
> bit, but maybe you know off the top of your head the right way to fix it.
>
> On Wed, Dec 2, 2015 at 11:32 AM Zachary Turner <zturner at google.com> wrote:
>
>> Oh yea, I made up that decorator idea because I didn't know all the
>> formatters were derived from a common base.  But your idea is better if
>> everything is derived from a common base.  To be honest you could even just
>> generate an error if there are two ResultsFormatter derived classes in the
>> same module.  We should be encouraging more smaller files with single
>> responsibility.  One of the things I plan to do as part of some cleanup in
>> a week or two is to split up dotest, dosep, and lldbtest.py into a couple
>> different files by breaking out things like TestBase, etc into separate
>> files.  So that it's easier to keep a mental map of where different code is.
>>
>> On Wed, Dec 2, 2015 at 11:26 AM Todd Fiala <todd.fiala at gmail.com> wrote:
>>
>>> On Wed, Dec 2, 2015 at 11:20 AM, Todd Fiala <todd.fiala at gmail.com>
>>> wrote:
>>>
>>>> Yeah I'd be good with that.  I can change that as well.
>>>>
>>>> -Todd
>>>>
>>>> On Wed, Dec 2, 2015 at 11:10 AM, Zachary Turner <zturner at google.com>
>>>> wrote:
>>>>
>>>>> Also another stylistic suggestion.  I've been thinking about how to
>>>>> more logically organize all the source files now that we have a package.
>>>>> So it makes sense conceptually to group all of the different result
>>>>> formatters under a subpackage called formatters.  So right now you've got
>>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter but it
>>>>> might make sense for this to be
>>>>> lldbsuite.test.formatters.basic.BasicResultsFormatter.  If you do things
>>>>> this way, it can actually result in a substantially shorter command line,
>>>>> because the --results-formatter option can use lldbsuite.test.formatters as
>>>>> a starting point.  So you could instead write:
>>>>>
>>>>> test/dotest.py --results-formatter basic
>>>>>
>>>>> dotest then looks for a `basic.py` module in the
>>>>> `lldbsuite.test.formatters` package, looks for a class inside with a
>>>>> @result_formatter decorator, and instantiates that.
>>>>>
>>>>> This has the advantage of making the command line shorter *and* a more
>>>>> logical source file organization.
>>>>>
>>>>
>>> The other thing that could allow me to do is possibly short-circuit the
>>> results formatter specifier so that, if just the module is specified, and
>>> if the module only has one ResultsFormatter-derived class, I can probably
>>> rig up code that figures out the right results formatter, shortening the
>>> required discriminator to something even shorter (i.e. module.classname
>>> becomes just module.)
>>>
>>>
>>>>
>>>>> On Wed, Dec 2, 2015 at 11:04 AM Zachary Turner <zturner at google.com>
>>>>> wrote:
>>>>>
>>>>>> Can --results-file=stdout be the default so that we don't have to
>>>>>> specify that?
>>>>>>
>>>>>> On Wed, Dec 2, 2015 at 11:02 AM Todd Fiala via lldb-dev <
>>>>>> lldb-dev at lists.llvm.org> wrote:
>>>>>>
>>>>>>> Also, all the text in the summary is fixed-width lined up nicely,
>>>>>>> which may not show in the commit message description if you're using a
>>>>>>> variable-width font.  On a terminal it looks nice.
>>>>>>>
>>>>>>> On Wed, Dec 2, 2015 at 11:01 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Dec 2, 2015 at 10:57 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I just put up an optional test results formatter that is a
>>>>>>>>> prototype of what we may move towards for our default test summary
>>>>>>>>> results.  It went in here:
>>>>>>>>>
>>>>>>>>> r254530
>>>>>>>>>
>>>>>>>>> and you can try it out with something like:
>>>>>>>>>
>>>>>>>>> time test/dotest.py --executable `pwd`/build/Debug/lldb
>>>>>>>>> --results-formatter
>>>>>>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file
>>>>>>>>> st
>>>>>>>>> out
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I cut and paste my line, but more than likely for most people you'd
>>>>>>>> just want this:
>>>>>>>>
>>>>>>>> test/dotest.py --results-formatter
>>>>>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file
>>>>>>>> stdout
>>>>>>>>
>>>>>>>> The other stuff was specific to my setup.  That line assumes you
>>>>>>>> run from the lldb source dir root.
>>>>>>>>
>>>>>>>>
>>>>>>>> Let me know if this satisfies the basic needs of counts and
>>>>>>>>> whatnot.  It counts test method runs rather than all the oddball "file,
>>>>>>>>> class, etc." counts we had before.
>>>>>>>>>
>>>>>>>>> It prints out the Details section when there are details, and
>>>>>>>>> keeps it nice and clean when there are none.
>>>>>>>>>
>>>>>>>>> It also mentions a bit about test reruns up top, but that won't
>>>>>>>>> come into play until I get the multi-test-pass, single-worker/low-load
>>>>>>>>> mechanism in place, which will depend on newer rerun count awareness
>>>>>>>>> support.
>>>>>>>>>
>>>>>>>>> The change also cleans up places where the test event framework
>>>>>>>>> was using string codes and replaces them with symbolic constants.
>>>>>>>>>
>>>>>>>>> Let me know what you think.  I can tweak it as needed to address
>>>>>>>>> testbot and other needs.  Once it looks reasonable, I'd like to move over
>>>>>>>>> to using it by default in the parallel test runner rather than the legacy
>>>>>>>>> support.
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>> --
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> -Todd
>>>>>>> _______________________________________________
>>>>>>> lldb-dev mailing list
>>>>>>> lldb-dev at lists.llvm.org
>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>> -Todd
>>>>
>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151202/a9ac841c/attachment.html>