[lldb-dev] Separating test runner and tests

[Todd Fiala via lldb-dev]

Unittest.

Comes with Python.

On Fri, Dec 11, 2015 at 11:07 AM, Zachary Turner <zturner at google.com> wrote:

> Presumably those tests use an entirely different, hand-rolled test running
> infrastructure?
>
> On Fri, Dec 11, 2015 at 10:52 AM Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> One thing I want to make sure we can do is have a sane way of storing and
>> running tests that  test the test execution engine.  Those are tests that
>> should not run as part of an "lldb test run".  These are tests that
>> maintainers of the test system run to make sure we're not breaking stuff
>> when we touch the test system.
>>
>> I would be writing more of those if I had a semi-sane way of doing it.
>>  (Part of the reason I broke out the python-based timeout logic the way I
>> did, before the major packaging changes, was so I had an obvious spot to
>> add tests for the process runner logic).
>>
>> On Fri, Dec 11, 2015 at 10:03 AM, Todd Fiala <todd.fiala at gmail.com>
>> wrote:
>>
>>> I like it.
>>>
>>> On Fri, Dec 11, 2015 at 9:51 AM, Zachary Turner <zturner at google.com>
>>> wrote:
>>>
>>>> Yea wasn't planning on doing this today, just throwing the idea out
>>>> there.
>>>>
>>>> On Fri, Dec 11, 2015 at 9:35 AM Todd Fiala <todd.fiala at gmail.com>
>>>> wrote:
>>>>
>>>>> I'm fine with the idea.
>>>>>
>>>>> FWIW the test events model will likely shift a bit, as it is currently
>>>>> a single sink, whereas I am likely to turn it into a test event filter
>>>>> chain shortly here.  Formatters still make sense as they'll be the things
>>>>> at the end of the chain.
>>>>>
>>>>> Minor detail, result_formatter.py should be results_formatter.py -
>>>>> they are ResultsFormatter instances (plural on Results since it transforms
>>>>> a series of results into coherent reported output).  I'll rename that at
>>>>> some point in the near future, but if you shift a number of things around,
>>>>> you can do that.
>>>>>
>>>>> I'm just about done with the multi-pass running.  I expect to get an
>>>>> opt-in version of that running end of day today or worst case on Sunday.
>>>>> It would be awesome if you can hold off on any significant change like that
>>>>> until this little bit is done as I'm sure we'll collide, particularly since
>>>>> this hits dosep.py pretty significantly.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> -Todd
>>>>>
>>>>> On Fri, Dec 11, 2015 at 1:33 AM, Pavel Labath via lldb-dev <
>>>>> lldb-dev at lists.llvm.org> wrote:
>>>>>
>>>>>> Sounds like a reasonable thing to do. A couple of tiny remarks:
>>>>>> - when you do the move, you might as well rename dotest into something
>>>>>> else, just to avoid the "which dotest should I run" type of
>>>>>> questions...
>>>>>> - there is nothing that makes it obvious that "engine" is actually a
>>>>>> "test running engine", as it sits in a sibling folder. OTOH,
>>>>>> "test_engine" might be too verbose, and messes up tab completion, so
>>>>>> that might not be a good idea either...
>>>>>>
>>>>>> pl
>>>>>>
>>>>>>
>>>>>> On 10 December 2015 at 23:30, Zachary Turner via lldb-dev
>>>>>> <lldb-dev at lists.llvm.org> wrote:
>>>>>> > Currently our folder structure looks like this:
>>>>>> >
>>>>>> > lldbsuite
>>>>>> > |-- test
>>>>>> >     |-- dotest.py
>>>>>> >     |-- dosep.py
>>>>>> >     |-- lldbtest.py
>>>>>> >     |-- ...
>>>>>> >     |-- functionalities
>>>>>> >     |-- lang
>>>>>> >     |-- expression_command
>>>>>> >     |-- ...
>>>>>> > etc
>>>>>> >
>>>>>> > I've been thinking about organizing it like this instead:
>>>>>> >
>>>>>> > lldbsuite
>>>>>> > |-- test
>>>>>> >     |-- functionalities
>>>>>> >     |-- lang
>>>>>> >     |-- expression_command
>>>>>> >     |-- ...
>>>>>> > |-- engine
>>>>>> >     |-- dotest.py
>>>>>> >     |-- dosep.py
>>>>>> >     |-- lldbtest.py
>>>>>> >     |-- ...
>>>>>> >
>>>>>> > Anybody have any thoughts on this?  Good idea or bad idea?  The
>>>>>> main reason
>>>>>> > I want to do this is because as we start breaking up some of the
>>>>>> code, it
>>>>>> > makes sense to start having some subpackages under the `engine`
>>>>>> folder (or
>>>>>> > the `test` folder in our current world).  For example, Todd and I
>>>>>> have
>>>>>> > discussed the idea of putting formatter related stuff under a
>>>>>> `formatters`
>>>>>> > subpackage.  In the current world, there's no way to differentiate
>>>>>> between
>>>>>> > folders which contain tests and folders which contain test
>>>>>> infrastructure,
>>>>>> > so when we walk the directory tree looking for tests we end up
>>>>>> walking a
>>>>>> > bunch of directories that are used for test infrastructure code and
>>>>>> not
>>>>>> > actual tests.  So I like the logical separation this provides --
>>>>>> having the
>>>>>> > tests themselves all under a single subpackage.
>>>>>> >
>>>>>> > Thoughts?
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > lldb-dev mailing list
>>>>>> > lldb-dev at lists.llvm.org
>>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>> >
>>>>>> _______________________________________________
>>>>>> lldb-dev mailing list
>>>>>> lldb-dev at lists.llvm.org
>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Todd
>>>>>
>>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
>>
>>
>> --
>> -Todd
>>
>


-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151211/8d3fcbaa/attachment.html>