[lldb-dev] Separating test runner and tests

Todd Fiala via lldb-dev lldb-dev at lists.llvm.org
Fri Dec 11 13:51:30 PST 2015 

Seems reasonable.

On Fri, Dec 11, 2015 at 11:46 AM, Zachary Turner <zturner at google.com> wrote:

> I think the correct fix is to just not throw the exception, and do it the
> right way.  unittest doesn't even use an exception for that anymore, but
> has some kind of method to tag the test as xfail or skip, whereas our
> unittest2 uses an exceptionf or the same purpose.
>
> Basically, we need to look at the docs for unittest, and update all of our
> code to only use methods / classes / etc that are publicly documented.
>
> On Fri, Dec 11, 2015 at 11:32 AM Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> Okay.  Sounds like something we can work around one way or another,
>> either by introducing the correct exception name for unittest, or
>> introducing our own if we need to do so.
>>
>> On Fri, Dec 11, 2015 at 11:22 AM, Zachary Turner <zturner at google.com>
>> wrote:
>>
>>> If I remember correctly it was in the way we had implemented one of the
>>> expected fail decorators.  We were manually throwing some kind of exception
>>> to indicate an xfail or a skip, and that exception doesn't exist in the
>>> upstream unittest.  Basically, we were relying on an implementation detail
>>> of unittest2
>>>
>>> On Fri, Dec 11, 2015 at 11:20 AM Todd Fiala <todd.fiala at gmail.com>
>>> wrote:
>>>
>>>> I think we can do this, and I'd like us to do this unless it's proven
>>>> to break something we're not aware of.  I think you did some research on
>>>> this after we discussed last, but something (maybe in the decorators)
>>>> didn't just work.  Was that right?
>>>>
>>>> On Fri, Dec 11, 2015 at 11:18 AM, Zachary Turner <zturner at google.com>
>>>> wrote:
>>>>
>>>>> Also at some point I will probably want to kill unittest2 and move to
>>>>> the upstream unittest.  AFAICT we only use unittest2 because it works on
>>>>> 2.6 and unittest doesn't.  But now that we're ok with saying 2.6 is
>>>>> unsupported, we can in theory go to the upstream unittest.
>>>>>
>>>>> On Fri, Dec 11, 2015 at 11:17 AM Zachary Turner <zturner at google.com>
>>>>> wrote:
>>>>>
>>>>>> Not sure I follow.  Are you trying to test the execution engine
>>>>>> itself (dotest.py, lldbtest.py, etc) or are you trying to have another
>>>>>> alternative to running individual tests?  The
>>>>>>
>>>>>> if __name__ == "__main__":
>>>>>>     unittest.main() stuff
>>>>>>
>>>>>> was deleted deleted from all tests a few months ago as part of the
>>>>>> package re-organization, and I thought I had general consensus at the time
>>>>>> that that was ok to do.
>>>>>>
>>>>>> On Fri, Dec 11, 2015 at 11:13 AM Todd Fiala <todd.fiala at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> It just requires running the test file as a python script.
>>>>>>>
>>>>>>> The runner is fired off like this:
>>>>>>>
>>>>>>> if __name__ == "__main__":
>>>>>>>     unittest.main()
>>>>>>>
>>>>>>> which is typically added to the bottom of all test files so you can
>>>>>>> call it directly.
>>>>>>>
>>>>>>> -Todd
>>>>>>>
>>>>>>> On Fri, Dec 11, 2015 at 11:12 AM, Todd Fiala <todd.fiala at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Unittest.
>>>>>>>>
>>>>>>>> Comes with Python.
>>>>>>>>
>>>>>>>> On Fri, Dec 11, 2015 at 11:07 AM, Zachary Turner <
>>>>>>>> zturner at google.com> wrote:
>>>>>>>>
>>>>>>>>> Presumably those tests use an entirely different, hand-rolled test
>>>>>>>>> running infrastructure?
>>>>>>>>>
>>>>>>>>> On Fri, Dec 11, 2015 at 10:52 AM Todd Fiala <todd.fiala at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> One thing I want to make sure we can do is have a sane way of
>>>>>>>>>> storing and running tests that  test the test execution engine.  Those are
>>>>>>>>>> tests that should not run as part of an "lldb test run".  These are tests
>>>>>>>>>> that maintainers of the test system run to make sure we're not breaking
>>>>>>>>>> stuff when we touch the test system.
>>>>>>>>>>
>>>>>>>>>> I would be writing more of those if I had a semi-sane way of
>>>>>>>>>> doing it.  (Part of the reason I broke out the python-based timeout logic
>>>>>>>>>> the way I did, before the major packaging changes, was so I had an obvious
>>>>>>>>>> spot to add tests for the process runner logic).
>>>>>>>>>>
>>>>>>>>>> On Fri, Dec 11, 2015 at 10:03 AM, Todd Fiala <
>>>>>>>>>> todd.fiala at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I like it.
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Dec 11, 2015 at 9:51 AM, Zachary Turner <
>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Yea wasn't planning on doing this today, just throwing the idea
>>>>>>>>>>>> out there.
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Dec 11, 2015 at 9:35 AM Todd Fiala <
>>>>>>>>>>>> todd.fiala at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I'm fine with the idea.
>>>>>>>>>>>>>
>>>>>>>>>>>>> FWIW the test events model will likely shift a bit, as it is
>>>>>>>>>>>>> currently a single sink, whereas I am likely to turn it into a test event
>>>>>>>>>>>>> filter chain shortly here.  Formatters still make sense as they'll be the
>>>>>>>>>>>>> things at the end of the chain.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Minor detail, result_formatter.py should be
>>>>>>>>>>>>> results_formatter.py - they are ResultsFormatter instances (plural on
>>>>>>>>>>>>> Results since it transforms a series of results into coherent reported
>>>>>>>>>>>>> output).  I'll rename that at some point in the near future, but if you
>>>>>>>>>>>>> shift a number of things around, you can do that.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm just about done with the multi-pass running.  I expect to
>>>>>>>>>>>>> get an opt-in version of that running end of day today or worst case on
>>>>>>>>>>>>> Sunday.  It would be awesome if you can hold off on any significant change
>>>>>>>>>>>>> like that until this little bit is done as I'm sure we'll collide,
>>>>>>>>>>>>> particularly since this hits dosep.py pretty significantly.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Dec 11, 2015 at 1:33 AM, Pavel Labath via lldb-dev <
>>>>>>>>>>>>> lldb-dev at lists.llvm.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sounds like a reasonable thing to do. A couple of tiny
>>>>>>>>>>>>>> remarks:
>>>>>>>>>>>>>> - when you do the move, you might as well rename dotest into
>>>>>>>>>>>>>> something
>>>>>>>>>>>>>> else, just to avoid the "which dotest should I run" type of
>>>>>>>>>>>>>> questions...
>>>>>>>>>>>>>> - there is nothing that makes it obvious that "engine" is
>>>>>>>>>>>>>> actually a
>>>>>>>>>>>>>> "test running engine", as it sits in a sibling folder. OTOH,
>>>>>>>>>>>>>> "test_engine" might be too verbose, and messes up tab
>>>>>>>>>>>>>> completion, so
>>>>>>>>>>>>>> that might not be a good idea either...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> pl
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 10 December 2015 at 23:30, Zachary Turner via lldb-dev
>>>>>>>>>>>>>> <lldb-dev at lists.llvm.org> wrote:
>>>>>>>>>>>>>> > Currently our folder structure looks like this:
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > lldbsuite
>>>>>>>>>>>>>> > |-- test
>>>>>>>>>>>>>> >     |-- dotest.py
>>>>>>>>>>>>>> >     |-- dosep.py
>>>>>>>>>>>>>> >     |-- lldbtest.py
>>>>>>>>>>>>>> >     |-- ...
>>>>>>>>>>>>>> >     |-- functionalities
>>>>>>>>>>>>>> >     |-- lang
>>>>>>>>>>>>>> >     |-- expression_command
>>>>>>>>>>>>>> >     |-- ...
>>>>>>>>>>>>>> > etc
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > I've been thinking about organizing it like this instead:
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > lldbsuite
>>>>>>>>>>>>>> > |-- test
>>>>>>>>>>>>>> >     |-- functionalities
>>>>>>>>>>>>>> >     |-- lang
>>>>>>>>>>>>>> >     |-- expression_command
>>>>>>>>>>>>>> >     |-- ...
>>>>>>>>>>>>>> > |-- engine
>>>>>>>>>>>>>> >     |-- dotest.py
>>>>>>>>>>>>>> >     |-- dosep.py
>>>>>>>>>>>>>> >     |-- lldbtest.py
>>>>>>>>>>>>>> >     |-- ...
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Anybody have any thoughts on this?  Good idea or bad idea?
>>>>>>>>>>>>>> The main reason
>>>>>>>>>>>>>> > I want to do this is because as we start breaking up some
>>>>>>>>>>>>>> of the code, it
>>>>>>>>>>>>>> > makes sense to start having some subpackages under the
>>>>>>>>>>>>>> `engine` folder (or
>>>>>>>>>>>>>> > the `test` folder in our current world).  For example, Todd
>>>>>>>>>>>>>> and I have
>>>>>>>>>>>>>> > discussed the idea of putting formatter related stuff under
>>>>>>>>>>>>>> a `formatters`
>>>>>>>>>>>>>> > subpackage.  In the current world, there's no way to
>>>>>>>>>>>>>> differentiate between
>>>>>>>>>>>>>> > folders which contain tests and folders which contain test
>>>>>>>>>>>>>> infrastructure,
>>>>>>>>>>>>>> > so when we walk the directory tree looking for tests we end
>>>>>>>>>>>>>> up walking a
>>>>>>>>>>>>>> > bunch of directories that are used for test infrastructure
>>>>>>>>>>>>>> code and not
>>>>>>>>>>>>>> > actual tests.  So I like the logical separation this
>>>>>>>>>>>>>> provides -- having the
>>>>>>>>>>>>>> > tests themselves all under a single subpackage.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Thoughts?
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>>>> > lldb-dev mailing list
>>>>>>>>>>>>>> > lldb-dev at lists.llvm.org
>>>>>>>>>>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> lldb-dev mailing list
>>>>>>>>>>>>>> lldb-dev at lists.llvm.org
>>>>>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> -Todd
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> -Todd
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> -Todd
>>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>> -Todd
>>>>
>>>
>>
>>
>> --
>> -Todd
>>
>


-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151211/9cbfcd99/attachment.html>

More information about the lldb-dev mailing list