[lldb-dev] unit testing C++ code in LLDB

[Todd Fiala]

> Why not just make it a python test?

I think I see the usefulness for it.  You really want to test a C++ class
at a low level and make sure it's working right.  But the state machine
needed to feed it inputs and outputs is complex enough that it would take a
lot of code to set that up right.  And you want it to always reflect what
lldb is doing, not some non-real-world static test environment where it can
get out of sync with the real lldb code.

-Todd

On Fri, Oct 3, 2014 at 11:24 AM, Zachary Turner <zturner at google.com> wrote:

> I think it diminishes their usefulness if they're only available to people
> willing to run them a specific way.  The python support on Windows isn't as
> rosy as it is on other platforms, and it's still very difficult to build
> LLDB with python support on Windows.  I might be the only person doing it.
> I'm trying to improve it, but I don't see it being in the same place as it
> is on other platforms for a while.
>
> Even ignoring that though, I think if your test needs to do setup in
> python, it should just be a regular python test of the public API like
> everything else.  Regardless, the functionality available to you from C++
> is a superset of that available to you from python.  You can even use the
> actual public API from C++, which is the same as what you'd be doing in
> python.  If you actually need to piggyback off of lots of already-written
> python code, then I'm wondering why this particular test is better suited
> for a gtest.  Why not just make it a python test?
>
> On Fri, Oct 3, 2014 at 11:01 AM, Sean Callanan <scallanan at apple.com>
> wrote:
>
>> Zach,
>>
>> I can live with two entry points – one without the Python dependency, one
>> accessible through Python.  As you (and Greg, in the past) suggest, we can
>> have a special public API for running unit tests – probably only in debug
>> builds – and use that API from Python.
>>
>> I’m not sure that all internal unit tests should do their setup in C++.
>> I think it makes the test more fragile – and wastes a lot of the machinery
>> we already have – to write a bunch of process-control logic in C++ when
>> what I actually want to test is something specific in an unrelated class.
>> LLDB is pretty closely tied to Python – for the test cases I write for the
>> expression parser, I think I’d be willing to mandate that Python be
>> available rather than make setup more challenging.
>>
>> So that both use cases can coexist, we can just make sure that both the
>> gtest runner and the SB API have the ability to run a subset of the unit
>> tests; the gtest runner runs all those that don’t require external setup,
>> and the SB API can select the tests that need to run with a specific
>> initial setup.
>>
>> Is that something that gtest would support?
>>
>> Sean
>>
>> On Oct 3, 2014, at 10:37 AM, Zachary Turner <zturner at google.com> wrote:
>>
>> I don't think the unit tests should depend on the python tests.  They
>> should be self contained.  In other words, the unit tests must be useful to
>> someone who is compiling without support for embedded python.  I wouldn't
>> want to have a unit test which is only useful if it's called from Python
>> which has already done some initial setup.  Still, if you want to avoid
>> having another entry poitn for convenience, you could expose something from
>> the public API that allows you to just say "run all the unittests".  But
>> there shouldn't be any setup in the python.  All the setup necessary to run
>> a given test should happen in C++.
>>
>> On Fri, Oct 3, 2014 at 10:23 AM, Sean Callanan <scallanan at apple.com>
>> wrote:
>>
>>>
>>> > On Oct 2, 2014, at 9:27 PM, Todd Fiala <tfiala at google.com> wrote:
>>> >
>>> > Hey Sean!
>>> > …
>>>
>>> Thanks for the introduction!  It looks like this is definitely in the
>>> direction of what I want.
>>>
>>> > If we want to do collaboration tests (integration tests, etc.), we're
>>> probably into the "should be in python category", but we might have a few
>>> low-level multi-class testing scenarios where we might want a different
>>> gtest/functional, gtest/integration or something similar directory
>>> structure to handle those.  Would be good to have discussion around that if
>>> we find a valid use for it.
>>>
>>> One thing I would like to be able to do for the expression parser is
>>> unit test in the context of a stopped process.
>>> I’m thinking of scenarios where I’d like to test e.g. the Materializer’s
>>> ability to read in variable data and make correct ValueObjects.
>>>
>>> One way to achieve this that comes to mind is to have a hook into the
>>> unit tests from the Python test suite, so we can set up the program’s state
>>> appropriately using our normal Python apparatus and then exercise exactly
>>> the functionality we want.
>>>
>>> Once we’ve got that kind of hook, we could just run all unit tests right
>>> from the Python test suite and avoid having another entry point.
>>>
>>> If you want IDE-friendly output, you could have an IDE-level target that
>>> runs test/dotest.py but singles out the unit tests.
>>>
>>> What do you think?
>>>
>>> Sean
>>>
>>
>>
>>
>


-- 
Todd Fiala | Software Engineer | tfiala at google.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20141003/c224c3c6/attachment.html>