[lldb-dev] Separating test runner and tests

[Zachary Turner via lldb-dev]

Presumably those tests use an entirely different, hand-rolled test running
infrastructure?

On Fri, Dec 11, 2015 at 10:52 AM Todd Fiala <todd.fiala at gmail.com> wrote:

> One thing I want to make sure we can do is have a sane way of storing and
> running tests that  test the test execution engine.  Those are tests that
> should not run as part of an "lldb test run".  These are tests that
> maintainers of the test system run to make sure we're not breaking stuff
> when we touch the test system.
>
> I would be writing more of those if I had a semi-sane way of doing it.
>  (Part of the reason I broke out the python-based timeout logic the way I
> did, before the major packaging changes, was so I had an obvious spot to
> add tests for the process runner logic).
>
> On Fri, Dec 11, 2015 at 10:03 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> I like it.
>>
>> On Fri, Dec 11, 2015 at 9:51 AM, Zachary Turner <zturner at google.com>
>> wrote:
>>
>>> Yea wasn't planning on doing this today, just throwing the idea out
>>> there.
>>>
>>> On Fri, Dec 11, 2015 at 9:35 AM Todd Fiala <todd.fiala at gmail.com> wrote:
>>>
>>>> I'm fine with the idea.
>>>>
>>>> FWIW the test events model will likely shift a bit, as it is currently
>>>> a single sink, whereas I am likely to turn it into a test event filter
>>>> chain shortly here.  Formatters still make sense as they'll be the things
>>>> at the end of the chain.
>>>>
>>>> Minor detail, result_formatter.py should be results_formatter.py - they
>>>> are ResultsFormatter instances (plural on Results since it transforms a
>>>> series of results into coherent reported output).  I'll rename that at some
>>>> point in the near future, but if you shift a number of things around, you
>>>> can do that.
>>>>
>>>> I'm just about done with the multi-pass running.  I expect to get an
>>>> opt-in version of that running end of day today or worst case on Sunday.
>>>> It would be awesome if you can hold off on any significant change like that
>>>> until this little bit is done as I'm sure we'll collide, particularly since
>>>> this hits dosep.py pretty significantly.
>>>>
>>>> Thanks!
>>>>
>>>> -Todd
>>>>
>>>> On Fri, Dec 11, 2015 at 1:33 AM, Pavel Labath via lldb-dev <
>>>> lldb-dev at lists.llvm.org> wrote:
>>>>
>>>>> Sounds like a reasonable thing to do. A couple of tiny remarks:
>>>>> - when you do the move, you might as well rename dotest into something
>>>>> else, just to avoid the "which dotest should I run" type of
>>>>> questions...
>>>>> - there is nothing that makes it obvious that "engine" is actually a
>>>>> "test running engine", as it sits in a sibling folder. OTOH,
>>>>> "test_engine" might be too verbose, and messes up tab completion, so
>>>>> that might not be a good idea either...
>>>>>
>>>>> pl
>>>>>
>>>>>
>>>>> On 10 December 2015 at 23:30, Zachary Turner via lldb-dev
>>>>> <lldb-dev at lists.llvm.org> wrote:
>>>>> > Currently our folder structure looks like this:
>>>>> >
>>>>> > lldbsuite
>>>>> > |-- test
>>>>> >     |-- dotest.py
>>>>> >     |-- dosep.py
>>>>> >     |-- lldbtest.py
>>>>> >     |-- ...
>>>>> >     |-- functionalities
>>>>> >     |-- lang
>>>>> >     |-- expression_command
>>>>> >     |-- ...
>>>>> > etc
>>>>> >
>>>>> > I've been thinking about organizing it like this instead:
>>>>> >
>>>>> > lldbsuite
>>>>> > |-- test
>>>>> >     |-- functionalities
>>>>> >     |-- lang
>>>>> >     |-- expression_command
>>>>> >     |-- ...
>>>>> > |-- engine
>>>>> >     |-- dotest.py
>>>>> >     |-- dosep.py
>>>>> >     |-- lldbtest.py
>>>>> >     |-- ...
>>>>> >
>>>>> > Anybody have any thoughts on this?  Good idea or bad idea?  The main
>>>>> reason
>>>>> > I want to do this is because as we start breaking up some of the
>>>>> code, it
>>>>> > makes sense to start having some subpackages under the `engine`
>>>>> folder (or
>>>>> > the `test` folder in our current world).  For example, Todd and I
>>>>> have
>>>>> > discussed the idea of putting formatter related stuff under a
>>>>> `formatters`
>>>>> > subpackage.  In the current world, there's no way to differentiate
>>>>> between
>>>>> > folders which contain tests and folders which contain test
>>>>> infrastructure,
>>>>> > so when we walk the directory tree looking for tests we end up
>>>>> walking a
>>>>> > bunch of directories that are used for test infrastructure code and
>>>>> not
>>>>> > actual tests.  So I like the logical separation this provides --
>>>>> having the
>>>>> > tests themselves all under a single subpackage.
>>>>> >
>>>>> > Thoughts?
>>>>> >
>>>>> > _______________________________________________
>>>>> > lldb-dev mailing list
>>>>> > lldb-dev at lists.llvm.org
>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>> >
>>>>> _______________________________________________
>>>>> lldb-dev mailing list
>>>>> lldb-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> -Todd
>>>>
>>>
>>
>>
>> --
>> -Todd
>>
>
>
>
> --
> -Todd
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151211/f465064a/attachment-0001.html>