[lldb-dev] Separating test runner and tests

Todd Fiala via lldb-dev lldb-dev at lists.llvm.org
Fri Dec 11 11:18:52 PST 2015 

On Fri, Dec 11, 2015 at 11:17 AM, Zachary Turner <zturner at google.com> wrote:

> Not sure I follow.  Are you trying to test the execution engine itself
> (dotest.py, lldbtest.py, etc)
>

This.  Test the lldb highly-specialized test runner internals.


> or are you trying to have another alternative to running individual tests?
>

Not this.


> The
>
> if __name__ == "__main__":
>     unittest.main() stuff
>
> was deleted deleted from all tests a few months ago as part of the package
> re-organization, and I thought I had general consensus at the time that
> that was ok to do.
>
>
Totally fine for lldb test suite tests, of which almost everything is.


> On Fri, Dec 11, 2015 at 11:13 AM Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> It just requires running the test file as a python script.
>>
>> The runner is fired off like this:
>>
>> if __name__ == "__main__":
>>     unittest.main()
>>
>> which is typically added to the bottom of all test files so you can call
>> it directly.
>>
>> -Todd
>>
>> On Fri, Dec 11, 2015 at 11:12 AM, Todd Fiala <todd.fiala at gmail.com>
>> wrote:
>>
>>> Unittest.
>>>
>>> Comes with Python.
>>>
>>> On Fri, Dec 11, 2015 at 11:07 AM, Zachary Turner <zturner at google.com>
>>> wrote:
>>>
>>>> Presumably those tests use an entirely different, hand-rolled test
>>>> running infrastructure?
>>>>
>>>> On Fri, Dec 11, 2015 at 10:52 AM Todd Fiala <todd.fiala at gmail.com>
>>>> wrote:
>>>>
>>>>> One thing I want to make sure we can do is have a sane way of storing
>>>>> and running tests that  test the test execution engine.  Those are tests
>>>>> that should not run as part of an "lldb test run".  These are tests that
>>>>> maintainers of the test system run to make sure we're not breaking stuff
>>>>> when we touch the test system.
>>>>>
>>>>> I would be writing more of those if I had a semi-sane way of doing it.
>>>>>  (Part of the reason I broke out the python-based timeout logic the way I
>>>>> did, before the major packaging changes, was so I had an obvious spot to
>>>>> add tests for the process runner logic).
>>>>>
>>>>> On Fri, Dec 11, 2015 at 10:03 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I like it.
>>>>>>
>>>>>> On Fri, Dec 11, 2015 at 9:51 AM, Zachary Turner <zturner at google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yea wasn't planning on doing this today, just throwing the idea out
>>>>>>> there.
>>>>>>>
>>>>>>> On Fri, Dec 11, 2015 at 9:35 AM Todd Fiala <todd.fiala at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I'm fine with the idea.
>>>>>>>>
>>>>>>>> FWIW the test events model will likely shift a bit, as it is
>>>>>>>> currently a single sink, whereas I am likely to turn it into a test event
>>>>>>>> filter chain shortly here.  Formatters still make sense as they'll be the
>>>>>>>> things at the end of the chain.
>>>>>>>>
>>>>>>>> Minor detail, result_formatter.py should be results_formatter.py -
>>>>>>>> they are ResultsFormatter instances (plural on Results since it transforms
>>>>>>>> a series of results into coherent reported output).  I'll rename that at
>>>>>>>> some point in the near future, but if you shift a number of things around,
>>>>>>>> you can do that.
>>>>>>>>
>>>>>>>> I'm just about done with the multi-pass running.  I expect to get
>>>>>>>> an opt-in version of that running end of day today or worst case on
>>>>>>>> Sunday.  It would be awesome if you can hold off on any significant change
>>>>>>>> like that until this little bit is done as I'm sure we'll collide,
>>>>>>>> particularly since this hits dosep.py pretty significantly.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>> On Fri, Dec 11, 2015 at 1:33 AM, Pavel Labath via lldb-dev <
>>>>>>>> lldb-dev at lists.llvm.org> wrote:
>>>>>>>>
>>>>>>>>> Sounds like a reasonable thing to do. A couple of tiny remarks:
>>>>>>>>> - when you do the move, you might as well rename dotest into
>>>>>>>>> something
>>>>>>>>> else, just to avoid the "which dotest should I run" type of
>>>>>>>>> questions...
>>>>>>>>> - there is nothing that makes it obvious that "engine" is actually
>>>>>>>>> a
>>>>>>>>> "test running engine", as it sits in a sibling folder. OTOH,
>>>>>>>>> "test_engine" might be too verbose, and messes up tab completion,
>>>>>>>>> so
>>>>>>>>> that might not be a good idea either...
>>>>>>>>>
>>>>>>>>> pl
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 10 December 2015 at 23:30, Zachary Turner via lldb-dev
>>>>>>>>> <lldb-dev at lists.llvm.org> wrote:
>>>>>>>>> > Currently our folder structure looks like this:
>>>>>>>>> >
>>>>>>>>> > lldbsuite
>>>>>>>>> > |-- test
>>>>>>>>> >     |-- dotest.py
>>>>>>>>> >     |-- dosep.py
>>>>>>>>> >     |-- lldbtest.py
>>>>>>>>> >     |-- ...
>>>>>>>>> >     |-- functionalities
>>>>>>>>> >     |-- lang
>>>>>>>>> >     |-- expression_command
>>>>>>>>> >     |-- ...
>>>>>>>>> > etc
>>>>>>>>> >
>>>>>>>>> > I've been thinking about organizing it like this instead:
>>>>>>>>> >
>>>>>>>>> > lldbsuite
>>>>>>>>> > |-- test
>>>>>>>>> >     |-- functionalities
>>>>>>>>> >     |-- lang
>>>>>>>>> >     |-- expression_command
>>>>>>>>> >     |-- ...
>>>>>>>>> > |-- engine
>>>>>>>>> >     |-- dotest.py
>>>>>>>>> >     |-- dosep.py
>>>>>>>>> >     |-- lldbtest.py
>>>>>>>>> >     |-- ...
>>>>>>>>> >
>>>>>>>>> > Anybody have any thoughts on this?  Good idea or bad idea?  The
>>>>>>>>> main reason
>>>>>>>>> > I want to do this is because as we start breaking up some of the
>>>>>>>>> code, it
>>>>>>>>> > makes sense to start having some subpackages under the `engine`
>>>>>>>>> folder (or
>>>>>>>>> > the `test` folder in our current world).  For example, Todd and
>>>>>>>>> I have
>>>>>>>>> > discussed the idea of putting formatter related stuff under a
>>>>>>>>> `formatters`
>>>>>>>>> > subpackage.  In the current world, there's no way to
>>>>>>>>> differentiate between
>>>>>>>>> > folders which contain tests and folders which contain test
>>>>>>>>> infrastructure,
>>>>>>>>> > so when we walk the directory tree looking for tests we end up
>>>>>>>>> walking a
>>>>>>>>> > bunch of directories that are used for test infrastructure code
>>>>>>>>> and not
>>>>>>>>> > actual tests.  So I like the logical separation this provides --
>>>>>>>>> having the
>>>>>>>>> > tests themselves all under a single subpackage.
>>>>>>>>> >
>>>>>>>>> > Thoughts?
>>>>>>>>> >
>>>>>>>>> > _______________________________________________
>>>>>>>>> > lldb-dev mailing list
>>>>>>>>> > lldb-dev at lists.llvm.org
>>>>>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>>> >
>>>>>>>>> _______________________________________________
>>>>>>>>> lldb-dev mailing list
>>>>>>>>> lldb-dev at lists.llvm.org
>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -Todd
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Todd
>>>>>
>>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
>>
>>
>> --
>> -Todd
>>
>


-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151211/63923dcf/attachment-0001.html>

More information about the lldb-dev mailing list