[lldb-dev] Separating test runner and tests

Todd Fiala via lldb-dev lldb-dev at lists.llvm.org
Fri Dec 11 11:15:57 PST 2015 

The tests end up looking substantially similar to our lldb test suite
tests, as they were based on unittest2, which is/was a relative of unittest
that now lives in Python.  The docs for unittest in python 2.x have
generally been accurate for the unittest2 lib we use.  At least, for the
areas I use.

On Fri, Dec 11, 2015 at 11:13 AM, Todd Fiala <todd.fiala at gmail.com> wrote:

> It just requires running the test file as a python script.
>
> The runner is fired off like this:
>
> if __name__ == "__main__":
>     unittest.main()
>
> which is typically added to the bottom of all test files so you can call
> it directly.
>
> -Todd
>
> On Fri, Dec 11, 2015 at 11:12 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> Unittest.
>>
>> Comes with Python.
>>
>> On Fri, Dec 11, 2015 at 11:07 AM, Zachary Turner <zturner at google.com>
>> wrote:
>>
>>> Presumably those tests use an entirely different, hand-rolled test
>>> running infrastructure?
>>>
>>> On Fri, Dec 11, 2015 at 10:52 AM Todd Fiala <todd.fiala at gmail.com>
>>> wrote:
>>>
>>>> One thing I want to make sure we can do is have a sane way of storing
>>>> and running tests that  test the test execution engine.  Those are tests
>>>> that should not run as part of an "lldb test run".  These are tests that
>>>> maintainers of the test system run to make sure we're not breaking stuff
>>>> when we touch the test system.
>>>>
>>>> I would be writing more of those if I had a semi-sane way of doing it.
>>>>  (Part of the reason I broke out the python-based timeout logic the way I
>>>> did, before the major packaging changes, was so I had an obvious spot to
>>>> add tests for the process runner logic).
>>>>
>>>> On Fri, Dec 11, 2015 at 10:03 AM, Todd Fiala <todd.fiala at gmail.com>
>>>> wrote:
>>>>
>>>>> I like it.
>>>>>
>>>>> On Fri, Dec 11, 2015 at 9:51 AM, Zachary Turner <zturner at google.com>
>>>>> wrote:
>>>>>
>>>>>> Yea wasn't planning on doing this today, just throwing the idea out
>>>>>> there.
>>>>>>
>>>>>> On Fri, Dec 11, 2015 at 9:35 AM Todd Fiala <todd.fiala at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I'm fine with the idea.
>>>>>>>
>>>>>>> FWIW the test events model will likely shift a bit, as it is
>>>>>>> currently a single sink, whereas I am likely to turn it into a test event
>>>>>>> filter chain shortly here.  Formatters still make sense as they'll be the
>>>>>>> things at the end of the chain.
>>>>>>>
>>>>>>> Minor detail, result_formatter.py should be results_formatter.py -
>>>>>>> they are ResultsFormatter instances (plural on Results since it transforms
>>>>>>> a series of results into coherent reported output).  I'll rename that at
>>>>>>> some point in the near future, but if you shift a number of things around,
>>>>>>> you can do that.
>>>>>>>
>>>>>>> I'm just about done with the multi-pass running.  I expect to get an
>>>>>>> opt-in version of that running end of day today or worst case on Sunday.
>>>>>>> It would be awesome if you can hold off on any significant change like that
>>>>>>> until this little bit is done as I'm sure we'll collide, particularly since
>>>>>>> this hits dosep.py pretty significantly.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> -Todd
>>>>>>>
>>>>>>> On Fri, Dec 11, 2015 at 1:33 AM, Pavel Labath via lldb-dev <
>>>>>>> lldb-dev at lists.llvm.org> wrote:
>>>>>>>
>>>>>>>> Sounds like a reasonable thing to do. A couple of tiny remarks:
>>>>>>>> - when you do the move, you might as well rename dotest into
>>>>>>>> something
>>>>>>>> else, just to avoid the "which dotest should I run" type of
>>>>>>>> questions...
>>>>>>>> - there is nothing that makes it obvious that "engine" is actually a
>>>>>>>> "test running engine", as it sits in a sibling folder. OTOH,
>>>>>>>> "test_engine" might be too verbose, and messes up tab completion, so
>>>>>>>> that might not be a good idea either...
>>>>>>>>
>>>>>>>> pl
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10 December 2015 at 23:30, Zachary Turner via lldb-dev
>>>>>>>> <lldb-dev at lists.llvm.org> wrote:
>>>>>>>> > Currently our folder structure looks like this:
>>>>>>>> >
>>>>>>>> > lldbsuite
>>>>>>>> > |-- test
>>>>>>>> >     |-- dotest.py
>>>>>>>> >     |-- dosep.py
>>>>>>>> >     |-- lldbtest.py
>>>>>>>> >     |-- ...
>>>>>>>> >     |-- functionalities
>>>>>>>> >     |-- lang
>>>>>>>> >     |-- expression_command
>>>>>>>> >     |-- ...
>>>>>>>> > etc
>>>>>>>> >
>>>>>>>> > I've been thinking about organizing it like this instead:
>>>>>>>> >
>>>>>>>> > lldbsuite
>>>>>>>> > |-- test
>>>>>>>> >     |-- functionalities
>>>>>>>> >     |-- lang
>>>>>>>> >     |-- expression_command
>>>>>>>> >     |-- ...
>>>>>>>> > |-- engine
>>>>>>>> >     |-- dotest.py
>>>>>>>> >     |-- dosep.py
>>>>>>>> >     |-- lldbtest.py
>>>>>>>> >     |-- ...
>>>>>>>> >
>>>>>>>> > Anybody have any thoughts on this?  Good idea or bad idea?  The
>>>>>>>> main reason
>>>>>>>> > I want to do this is because as we start breaking up some of the
>>>>>>>> code, it
>>>>>>>> > makes sense to start having some subpackages under the `engine`
>>>>>>>> folder (or
>>>>>>>> > the `test` folder in our current world).  For example, Todd and I
>>>>>>>> have
>>>>>>>> > discussed the idea of putting formatter related stuff under a
>>>>>>>> `formatters`
>>>>>>>> > subpackage.  In the current world, there's no way to
>>>>>>>> differentiate between
>>>>>>>> > folders which contain tests and folders which contain test
>>>>>>>> infrastructure,
>>>>>>>> > so when we walk the directory tree looking for tests we end up
>>>>>>>> walking a
>>>>>>>> > bunch of directories that are used for test infrastructure code
>>>>>>>> and not
>>>>>>>> > actual tests.  So I like the logical separation this provides --
>>>>>>>> having the
>>>>>>>> > tests themselves all under a single subpackage.
>>>>>>>> >
>>>>>>>> > Thoughts?
>>>>>>>> >
>>>>>>>> > _______________________________________________
>>>>>>>> > lldb-dev mailing list
>>>>>>>> > lldb-dev at lists.llvm.org
>>>>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>> >
>>>>>>>> _______________________________________________
>>>>>>>> lldb-dev mailing list
>>>>>>>> lldb-dev at lists.llvm.org
>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> -Todd
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Todd
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> -Todd
>>>>
>>>
>>
>>
>> --
>> -Todd
>>
>
>
>
> --
> -Todd
>



-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151211/2d441c64/attachment.html>

More information about the lldb-dev mailing list