[lldb-dev] Separating test runner and tests

Zachary Turner via lldb-dev lldb-dev at lists.llvm.org
Fri Dec 11 11:22:10 PST 2015 

If I remember correctly it was in the way we had implemented one of the
expected fail decorators.  We were manually throwing some kind of exception
to indicate an xfail or a skip, and that exception doesn't exist in the
upstream unittest.  Basically, we were relying on an implementation detail
of unittest2

On Fri, Dec 11, 2015 at 11:20 AM Todd Fiala <todd.fiala at gmail.com> wrote:

> I think we can do this, and I'd like us to do this unless it's proven to
> break something we're not aware of.  I think you did some research on this
> after we discussed last, but something (maybe in the decorators) didn't
> just work.  Was that right?
>
> On Fri, Dec 11, 2015 at 11:18 AM, Zachary Turner <zturner at google.com>
> wrote:
>
>> Also at some point I will probably want to kill unittest2 and move to the
>> upstream unittest.  AFAICT we only use unittest2 because it works on 2.6
>> and unittest doesn't.  But now that we're ok with saying 2.6 is
>> unsupported, we can in theory go to the upstream unittest.
>>
>> On Fri, Dec 11, 2015 at 11:17 AM Zachary Turner <zturner at google.com>
>> wrote:
>>
>>> Not sure I follow.  Are you trying to test the execution engine itself
>>> (dotest.py, lldbtest.py, etc) or are you trying to have another alternative
>>> to running individual tests?  The
>>>
>>> if __name__ == "__main__":
>>>     unittest.main() stuff
>>>
>>> was deleted deleted from all tests a few months ago as part of the
>>> package re-organization, and I thought I had general consensus at the time
>>> that that was ok to do.
>>>
>>> On Fri, Dec 11, 2015 at 11:13 AM Todd Fiala <todd.fiala at gmail.com>
>>> wrote:
>>>
>>>> It just requires running the test file as a python script.
>>>>
>>>> The runner is fired off like this:
>>>>
>>>> if __name__ == "__main__":
>>>>     unittest.main()
>>>>
>>>> which is typically added to the bottom of all test files so you can
>>>> call it directly.
>>>>
>>>> -Todd
>>>>
>>>> On Fri, Dec 11, 2015 at 11:12 AM, Todd Fiala <todd.fiala at gmail.com>
>>>> wrote:
>>>>
>>>>> Unittest.
>>>>>
>>>>> Comes with Python.
>>>>>
>>>>> On Fri, Dec 11, 2015 at 11:07 AM, Zachary Turner <zturner at google.com>
>>>>> wrote:
>>>>>
>>>>>> Presumably those tests use an entirely different, hand-rolled test
>>>>>> running infrastructure?
>>>>>>
>>>>>> On Fri, Dec 11, 2015 at 10:52 AM Todd Fiala <todd.fiala at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> One thing I want to make sure we can do is have a sane way of
>>>>>>> storing and running tests that  test the test execution engine.  Those are
>>>>>>> tests that should not run as part of an "lldb test run".  These are tests
>>>>>>> that maintainers of the test system run to make sure we're not breaking
>>>>>>> stuff when we touch the test system.
>>>>>>>
>>>>>>> I would be writing more of those if I had a semi-sane way of doing
>>>>>>> it.  (Part of the reason I broke out the python-based timeout logic the way
>>>>>>> I did, before the major packaging changes, was so I had an obvious spot to
>>>>>>> add tests for the process runner logic).
>>>>>>>
>>>>>>> On Fri, Dec 11, 2015 at 10:03 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I like it.
>>>>>>>>
>>>>>>>> On Fri, Dec 11, 2015 at 9:51 AM, Zachary Turner <zturner at google.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Yea wasn't planning on doing this today, just throwing the idea
>>>>>>>>> out there.
>>>>>>>>>
>>>>>>>>> On Fri, Dec 11, 2015 at 9:35 AM Todd Fiala <todd.fiala at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I'm fine with the idea.
>>>>>>>>>>
>>>>>>>>>> FWIW the test events model will likely shift a bit, as it is
>>>>>>>>>> currently a single sink, whereas I am likely to turn it into a test event
>>>>>>>>>> filter chain shortly here.  Formatters still make sense as they'll be the
>>>>>>>>>> things at the end of the chain.
>>>>>>>>>>
>>>>>>>>>> Minor detail, result_formatter.py should be results_formatter.py
>>>>>>>>>> - they are ResultsFormatter instances (plural on Results since it
>>>>>>>>>> transforms a series of results into coherent reported output).  I'll rename
>>>>>>>>>> that at some point in the near future, but if you shift a number of things
>>>>>>>>>> around, you can do that.
>>>>>>>>>>
>>>>>>>>>> I'm just about done with the multi-pass running.  I expect to get
>>>>>>>>>> an opt-in version of that running end of day today or worst case on
>>>>>>>>>> Sunday.  It would be awesome if you can hold off on any significant change
>>>>>>>>>> like that until this little bit is done as I'm sure we'll collide,
>>>>>>>>>> particularly since this hits dosep.py pretty significantly.
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> -Todd
>>>>>>>>>>
>>>>>>>>>> On Fri, Dec 11, 2015 at 1:33 AM, Pavel Labath via lldb-dev <
>>>>>>>>>> lldb-dev at lists.llvm.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Sounds like a reasonable thing to do. A couple of tiny remarks:
>>>>>>>>>>> - when you do the move, you might as well rename dotest into
>>>>>>>>>>> something
>>>>>>>>>>> else, just to avoid the "which dotest should I run" type of
>>>>>>>>>>> questions...
>>>>>>>>>>> - there is nothing that makes it obvious that "engine" is
>>>>>>>>>>> actually a
>>>>>>>>>>> "test running engine", as it sits in a sibling folder. OTOH,
>>>>>>>>>>> "test_engine" might be too verbose, and messes up tab
>>>>>>>>>>> completion, so
>>>>>>>>>>> that might not be a good idea either...
>>>>>>>>>>>
>>>>>>>>>>> pl
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 10 December 2015 at 23:30, Zachary Turner via lldb-dev
>>>>>>>>>>> <lldb-dev at lists.llvm.org> wrote:
>>>>>>>>>>> > Currently our folder structure looks like this:
>>>>>>>>>>> >
>>>>>>>>>>> > lldbsuite
>>>>>>>>>>> > |-- test
>>>>>>>>>>> >     |-- dotest.py
>>>>>>>>>>> >     |-- dosep.py
>>>>>>>>>>> >     |-- lldbtest.py
>>>>>>>>>>> >     |-- ...
>>>>>>>>>>> >     |-- functionalities
>>>>>>>>>>> >     |-- lang
>>>>>>>>>>> >     |-- expression_command
>>>>>>>>>>> >     |-- ...
>>>>>>>>>>> > etc
>>>>>>>>>>> >
>>>>>>>>>>> > I've been thinking about organizing it like this instead:
>>>>>>>>>>> >
>>>>>>>>>>> > lldbsuite
>>>>>>>>>>> > |-- test
>>>>>>>>>>> >     |-- functionalities
>>>>>>>>>>> >     |-- lang
>>>>>>>>>>> >     |-- expression_command
>>>>>>>>>>> >     |-- ...
>>>>>>>>>>> > |-- engine
>>>>>>>>>>> >     |-- dotest.py
>>>>>>>>>>> >     |-- dosep.py
>>>>>>>>>>> >     |-- lldbtest.py
>>>>>>>>>>> >     |-- ...
>>>>>>>>>>> >
>>>>>>>>>>> > Anybody have any thoughts on this?  Good idea or bad idea?
>>>>>>>>>>> The main reason
>>>>>>>>>>> > I want to do this is because as we start breaking up some of
>>>>>>>>>>> the code, it
>>>>>>>>>>> > makes sense to start having some subpackages under the
>>>>>>>>>>> `engine` folder (or
>>>>>>>>>>> > the `test` folder in our current world).  For example, Todd
>>>>>>>>>>> and I have
>>>>>>>>>>> > discussed the idea of putting formatter related stuff under a
>>>>>>>>>>> `formatters`
>>>>>>>>>>> > subpackage.  In the current world, there's no way to
>>>>>>>>>>> differentiate between
>>>>>>>>>>> > folders which contain tests and folders which contain test
>>>>>>>>>>> infrastructure,
>>>>>>>>>>> > so when we walk the directory tree looking for tests we end up
>>>>>>>>>>> walking a
>>>>>>>>>>> > bunch of directories that are used for test infrastructure
>>>>>>>>>>> code and not
>>>>>>>>>>> > actual tests.  So I like the logical separation this provides
>>>>>>>>>>> -- having the
>>>>>>>>>>> > tests themselves all under a single subpackage.
>>>>>>>>>>> >
>>>>>>>>>>> > Thoughts?
>>>>>>>>>>> >
>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>> > lldb-dev mailing list
>>>>>>>>>>> > lldb-dev at lists.llvm.org
>>>>>>>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>>>>> >
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> lldb-dev mailing list
>>>>>>>>>>> lldb-dev at lists.llvm.org
>>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> -Todd
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> -Todd
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Todd
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> -Todd
>>>>
>>>
>
>
> --
> -Todd
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151211/8e4f4692/attachment-0001.html>

More information about the lldb-dev mailing list