[lldb-dev] Separating test runner and tests

Todd Fiala via lldb-dev lldb-dev at lists.llvm.org
Fri Dec 11 11:20:01 PST 2015 

I think we can do this, and I'd like us to do this unless it's proven to
break something we're not aware of.  I think you did some research on this
after we discussed last, but something (maybe in the decorators) didn't
just work.  Was that right?

On Fri, Dec 11, 2015 at 11:18 AM, Zachary Turner <zturner at google.com> wrote:

> Also at some point I will probably want to kill unittest2 and move to the
> upstream unittest.  AFAICT we only use unittest2 because it works on 2.6
> and unittest doesn't.  But now that we're ok with saying 2.6 is
> unsupported, we can in theory go to the upstream unittest.
>
> On Fri, Dec 11, 2015 at 11:17 AM Zachary Turner <zturner at google.com>
> wrote:
>
>> Not sure I follow.  Are you trying to test the execution engine itself
>> (dotest.py, lldbtest.py, etc) or are you trying to have another alternative
>> to running individual tests?  The
>>
>> if __name__ == "__main__":
>>     unittest.main() stuff
>>
>> was deleted deleted from all tests a few months ago as part of the
>> package re-organization, and I thought I had general consensus at the time
>> that that was ok to do.
>>
>> On Fri, Dec 11, 2015 at 11:13 AM Todd Fiala <todd.fiala at gmail.com> wrote:
>>
>>> It just requires running the test file as a python script.
>>>
>>> The runner is fired off like this:
>>>
>>> if __name__ == "__main__":
>>>     unittest.main()
>>>
>>> which is typically added to the bottom of all test files so you can call
>>> it directly.
>>>
>>> -Todd
>>>
>>> On Fri, Dec 11, 2015 at 11:12 AM, Todd Fiala <todd.fiala at gmail.com>
>>> wrote:
>>>
>>>> Unittest.
>>>>
>>>> Comes with Python.
>>>>
>>>> On Fri, Dec 11, 2015 at 11:07 AM, Zachary Turner <zturner at google.com>
>>>> wrote:
>>>>
>>>>> Presumably those tests use an entirely different, hand-rolled test
>>>>> running infrastructure?
>>>>>
>>>>> On Fri, Dec 11, 2015 at 10:52 AM Todd Fiala <todd.fiala at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> One thing I want to make sure we can do is have a sane way of storing
>>>>>> and running tests that  test the test execution engine.  Those are tests
>>>>>> that should not run as part of an "lldb test run".  These are tests that
>>>>>> maintainers of the test system run to make sure we're not breaking stuff
>>>>>> when we touch the test system.
>>>>>>
>>>>>> I would be writing more of those if I had a semi-sane way of doing
>>>>>> it.  (Part of the reason I broke out the python-based timeout logic the way
>>>>>> I did, before the major packaging changes, was so I had an obvious spot to
>>>>>> add tests for the process runner logic).
>>>>>>
>>>>>> On Fri, Dec 11, 2015 at 10:03 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I like it.
>>>>>>>
>>>>>>> On Fri, Dec 11, 2015 at 9:51 AM, Zachary Turner <zturner at google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yea wasn't planning on doing this today, just throwing the idea out
>>>>>>>> there.
>>>>>>>>
>>>>>>>> On Fri, Dec 11, 2015 at 9:35 AM Todd Fiala <todd.fiala at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I'm fine with the idea.
>>>>>>>>>
>>>>>>>>> FWIW the test events model will likely shift a bit, as it is
>>>>>>>>> currently a single sink, whereas I am likely to turn it into a test event
>>>>>>>>> filter chain shortly here.  Formatters still make sense as they'll be the
>>>>>>>>> things at the end of the chain.
>>>>>>>>>
>>>>>>>>> Minor detail, result_formatter.py should be results_formatter.py -
>>>>>>>>> they are ResultsFormatter instances (plural on Results since it transforms
>>>>>>>>> a series of results into coherent reported output).  I'll rename that at
>>>>>>>>> some point in the near future, but if you shift a number of things around,
>>>>>>>>> you can do that.
>>>>>>>>>
>>>>>>>>> I'm just about done with the multi-pass running.  I expect to get
>>>>>>>>> an opt-in version of that running end of day today or worst case on
>>>>>>>>> Sunday.  It would be awesome if you can hold off on any significant change
>>>>>>>>> like that until this little bit is done as I'm sure we'll collide,
>>>>>>>>> particularly since this hits dosep.py pretty significantly.
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>> On Fri, Dec 11, 2015 at 1:33 AM, Pavel Labath via lldb-dev <
>>>>>>>>> lldb-dev at lists.llvm.org> wrote:
>>>>>>>>>
>>>>>>>>>> Sounds like a reasonable thing to do. A couple of tiny remarks:
>>>>>>>>>> - when you do the move, you might as well rename dotest into
>>>>>>>>>> something
>>>>>>>>>> else, just to avoid the "which dotest should I run" type of
>>>>>>>>>> questions...
>>>>>>>>>> - there is nothing that makes it obvious that "engine" is
>>>>>>>>>> actually a
>>>>>>>>>> "test running engine", as it sits in a sibling folder. OTOH,
>>>>>>>>>> "test_engine" might be too verbose, and messes up tab completion,
>>>>>>>>>> so
>>>>>>>>>> that might not be a good idea either...
>>>>>>>>>>
>>>>>>>>>> pl
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 10 December 2015 at 23:30, Zachary Turner via lldb-dev
>>>>>>>>>> <lldb-dev at lists.llvm.org> wrote:
>>>>>>>>>> > Currently our folder structure looks like this:
>>>>>>>>>> >
>>>>>>>>>> > lldbsuite
>>>>>>>>>> > |-- test
>>>>>>>>>> >     |-- dotest.py
>>>>>>>>>> >     |-- dosep.py
>>>>>>>>>> >     |-- lldbtest.py
>>>>>>>>>> >     |-- ...
>>>>>>>>>> >     |-- functionalities
>>>>>>>>>> >     |-- lang
>>>>>>>>>> >     |-- expression_command
>>>>>>>>>> >     |-- ...
>>>>>>>>>> > etc
>>>>>>>>>> >
>>>>>>>>>> > I've been thinking about organizing it like this instead:
>>>>>>>>>> >
>>>>>>>>>> > lldbsuite
>>>>>>>>>> > |-- test
>>>>>>>>>> >     |-- functionalities
>>>>>>>>>> >     |-- lang
>>>>>>>>>> >     |-- expression_command
>>>>>>>>>> >     |-- ...
>>>>>>>>>> > |-- engine
>>>>>>>>>> >     |-- dotest.py
>>>>>>>>>> >     |-- dosep.py
>>>>>>>>>> >     |-- lldbtest.py
>>>>>>>>>> >     |-- ...
>>>>>>>>>> >
>>>>>>>>>> > Anybody have any thoughts on this?  Good idea or bad idea?  The
>>>>>>>>>> main reason
>>>>>>>>>> > I want to do this is because as we start breaking up some of
>>>>>>>>>> the code, it
>>>>>>>>>> > makes sense to start having some subpackages under the `engine`
>>>>>>>>>> folder (or
>>>>>>>>>> > the `test` folder in our current world).  For example, Todd and
>>>>>>>>>> I have
>>>>>>>>>> > discussed the idea of putting formatter related stuff under a
>>>>>>>>>> `formatters`
>>>>>>>>>> > subpackage.  In the current world, there's no way to
>>>>>>>>>> differentiate between
>>>>>>>>>> > folders which contain tests and folders which contain test
>>>>>>>>>> infrastructure,
>>>>>>>>>> > so when we walk the directory tree looking for tests we end up
>>>>>>>>>> walking a
>>>>>>>>>> > bunch of directories that are used for test infrastructure code
>>>>>>>>>> and not
>>>>>>>>>> > actual tests.  So I like the logical separation this provides
>>>>>>>>>> -- having the
>>>>>>>>>> > tests themselves all under a single subpackage.
>>>>>>>>>> >
>>>>>>>>>> > Thoughts?
>>>>>>>>>> >
>>>>>>>>>> > _______________________________________________
>>>>>>>>>> > lldb-dev mailing list
>>>>>>>>>> > lldb-dev at lists.llvm.org
>>>>>>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>>>> >
>>>>>>>>>> _______________________________________________
>>>>>>>>>> lldb-dev mailing list
>>>>>>>>>> lldb-dev at lists.llvm.org
>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> -Todd
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -Todd
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> -Todd
>>>>
>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>


-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151211/4e91bf39/attachment.html>

More information about the lldb-dev mailing list