[lldb-dev] unit testing C++ code in LLDB

Todd Fiala tfiala at google.com
Thu Oct 2 21:51:52 PDT 2014 

The other bit to know about it is my first test subject
(ThreadStateCoordinator) is entirely not dependent on anything other than
standard C++, so I didn't need to pull in a ton of lldb code to make that
work.  In all likelihood (e.g. with expression parser) we'll need to grab
something like the lldb-core output or something, so a bit more make logic
will need to be worked out.

The do-gtest.py script right now just scans for directories with a
Makefile, and executes make in those directories.  It always runs a make
clean after a make.  If it is called with a 'clean' command-line arg, it
will just do the clean step and skip the make.  The do-gtest.py script
skips output from the clean step, but will reflect stdout/stderr from the
normal make step.  If any of the makes fail, the do-gtest.py reports the
exit failure code from the most recent failing make execution but continues
until all have been called.

-Todd


On Thu, Oct 2, 2014 at 9:27 PM, Todd Fiala <tfiala at google.com> wrote:

> Hey Sean!
>
> Yeah I've got a gtest subproject in lldb now.  It's not wired into any
> main test process (i.e. check-lldb or test targets don't run it, but if you
> run the "gtest" project in the lldb workspace in Xcode, or go to the gtest
> dir and run "do-gtest.py", you'll get it running).
>
> I've got it rigged up to put Xcode error markers at the point in gtests
> that fail at the failing assertion point after doing some output munging to
> make it work.
>
> The actual gtest build/runner works on MacOSX and Linux.  It *should* work
> on all platforms since I'm actually using the llvm-embedded gtest
> framework, so technically if llvm builds, so should the gtest setup.
>
> However, I do some weak (and need to be improved) inference of directory
> structure/build structure to find some necessary files in the
> gtest/make/Makefile.rules.  They can be overridden by environment or make
> variable settings, but they're definitely not going to work for everyone
> and there are probably more robust methods for figuring out certain
> directory locations.
>
> I run it fine on MacOSX with the canonical Xcode directory layout and
> using the Xcode Preferences | Locations set to put project output in a
> project-relative DerivedData directory.
>
> On Linux, I assume the build directory is a sibling directory to a llvm,
> llvm/tools/clang, llvm/tools/lldb directory structure and (here's the
> not-reasonable-default) I assume the build directory is called build-debug.
>  (So, I have a build-drectory alongside the top-level llvm directory).
> Again, all these can be overridden by environment variable/make variable
> settings, and I think I have the Makefile emit an error if it can't resolve
> the directories it uses and tells what variable to override to fix it.
> But, likely it needs some love.
>
> For me, the source/Plugins/Process/Linux/ThreadStateCoordinator.{h/cpp}
> was my guinea pig.  I'm using that to replace some guts in the llgs signal
> state transition logic that requires putting threads through some ptrace
> state magic before it should tell lldb about some thread stops.  And so I'm
> testing that all the deferred callbacks (i.e. essentially GCD-like
> constructs) fire off in the right circumstances.  That's an example of a
> small piece of code in C++ that I just need to know is working right and so
> I introduced the gtest framework around it.
>
> Hope that gets you started!  Feel free to improve and/or offer suggestions.
>
> For now, we had settled on using gtest/unittest as the root of 1-1
> correspondence directory structure mirroring for testing single classes.
>
> If we want to do collaboration tests (integration tests, etc.), we're
> probably into the "should be in python category", but we might have a few
> low-level multi-class testing scenarios where we might want a different
> gtest/functional, gtest/integration or something similar directory
> structure to handle those.  Would be good to have discussion around that if
> we find a valid use for it.
>
> -Todd
>
> On Thu, Oct 2, 2014 at 7:02 PM, Sean Callanan <scallanan at apple.com> wrote:
>
>> Waaaaitasecond, I’m seeing the gtest subproject…
>> it’s a bit embarrassing that I haven’t checked out top of tree in so long!
>> I’m going to take a look around.  Still interested in collaborating on
>> this.
>>
>> Sean
>>
>> On Oct 2, 2014, at 6:11 PM, Zachary Turner <zturner at google.com> wrote:
>>
>> I gave it a brief look, but I didn't get it working.  Clang and LLD both
>> already do this, so I figured it would be as simple as copying their CMake
>> for the gtest stuff and fixing it up with LLDB paths.  Unfortunately, I was
>> getting tons of linker errors, and I'm not really sure why, as there didn't
>> appear to be much magic there.  I didn't investigate it further after that,
>> but it would definitely be a good thing to tackle.
>>
>> On Thu, Oct 2, 2014 at 2:01 PM, Sean Callanan <scallanan at apple.com>
>> wrote:
>>
>>> At risk of being burnt at the stake for practicing the the necromantic
>>> arts:
>>> Jim’s point that internal APIs are somewhat fragile is well taken, and I
>>> don’t think there’s much value in using internal tests as “more convenient”
>>> substitutes for external tests.
>>>
>>> That said, I think internal unit tests add several benefits that improve
>>> code:
>>>
>>>
>>>    - *Modular design*.  If clients of an internal object have to do a
>>>    lot of extra work to make the object work properly, this may indicate a
>>>    design issue.
>>>    - *State encapsulation*.  If an object changes behaviors depending
>>>    on outside state, then that makes it much harder to use.
>>>    - *Code readability*.  Test cases can demonstrate how an object is
>>>    intended to be used, and act as compelling witnesses that that use case
>>>    works.
>>>
>>>
>>> I would be interested in using the expression parser as a guinea pig to
>>> introduce test-driven methods.
>>> Todd/Zach, did you ever get a gtest-based unit test system working?
>>>
>>> Sean
>>>
>>> On Jul 18, 2014, at 6:14 PM, Zachary Turner <zturner at google.com> wrote:
>>>
>>> FWIW I'm kind of in favor of bringing in gtest with limited use.  When I
>>> first started digging into LLDB, probably the first 3-4 bugs I fixed were
>>> all in IRMemoryMap, and they would all have been caught if the class were
>>> unit tested properly.
>>>
>>>
>>> On Fri, Jul 18, 2014 at 4:03 PM, Greg Clayton <gclayton at apple.com>
>>> wrote:
>>>
>>>> We could expose a new static function in SBDebugger:
>>>>
>>>> class SBDebugger {
>>>>
>>>> static void
>>>> UnitTest (const char *args);
>>>>
>>>> };
>>>>
>>>> Then internally it can parse any arguments and run tests as needed?
>>>> Each class that wants to register unit tests with the debugger during
>>>> SBDebugger::Initialize() which must be called prior to doing anything with
>>>> the LLDB API:
>>>>
>>>>
>>>> SBDebugger::Initialize()
>>>> {
>>>>     Debugger::Intialize();
>>>> }
>>>>
>>>> Debugger::Initialize()
>>>> {
>>>>         ....
>>>>         NamedPipe:: Initialize();
>>>> }
>>>>
>>>>
>>>> Then in the static NamedPipe::Initiazize() you could register your unit
>>>> tests:
>>>>
>>>> void
>>>> NamedPipe::Initialiaze()
>>>> {
>>>>     Debugger::RegisterUnitTest(NamedPipe::InitTest1,
>>>> "NamedPipe::InitTest1"));
>>>> }
>>>>
>>>>
>>>> Then you could just run:
>>>>
>>>> SBDebugger::Initialize();
>>>> SBDebugger::UnitTest(NULL); // Run all tests
>>>>
>>>> Or run individual ones:
>>>>
>>>> SBDebugger::Initialize();
>>>> SBDebugger::UnitTest("--test NamedPipe::InitTest1"); // Run just
>>>> "NamedPipe::InitTest1"
>>>>
>>>> Of course then the LLDB test suite could run these unit tests first to
>>>> make sure everything is good.
>>>>
>>>>
>>>>
>>>> > On Jul 16, 2014, at 3:59 PM, Zachary Turner <zturner at google.com>
>>>> wrote:
>>>> >
>>>> > If it makes you feel any better LLVM is leery of it too, and it's
>>>> only used, as you said, in specialized circumstances.  It's especially
>>>> useful for testing abstract data types, where you just want to test the
>>>> interface to a self-contained, reusable class.
>>>> >
>>>> >
>>>> > On Wed, Jul 16, 2014 at 2:25 PM, <jingham at apple.com> wrote:
>>>> > I'm a little leery about this.  We don't test at the lldb_private
>>>> layer because those tests are likely to be fragile and easily broken.  For
>>>> utility classes like NamedPipe I guess I don't so much mind, but I'm not
>>>> sure its a great idea to do this more generally.
>>>> >
>>>> > Jim
>>>> >
>>>> > > On Jul 16, 2014, at 9:40 AM, Todd Fiala <tfiala at google.com> wrote:
>>>> > >
>>>> > > Hey guys,
>>>> > >
>>>> > > Sometimes I have smaller bits of code I'd like to test in LLDB as
>>>> I'm developing them (i.e. TDD-style) that are C++ and won't be exposed
>>>> directly via Python.  I'm not sure I've seen any facilities in the LLDB
>>>> tests for adding such tests.  Essentially I'd want to do something like a
>>>> gtest or cppunit test.
>>>> > >
>>>> > > Do we have any mechanism for doing that currently?  If we do, what
>>>> is it?  If we don't, how about adding some mechanism to do it after we
>>>> figure out how we'd like to approach it?  Or, if you have thoughts on a
>>>> good, simple way to do it from Python that doesn't require extra Python
>>>> bindings just to do it, that'd be fine by me as well.
>>>> > >
>>>> > > If we want to take a concrete example, here is one: I'm adding a
>>>> NamedPipe class under the host dir.  I'd like to make some simple tests for
>>>> it, and test it under Linux, Windows and MacOSX.  In the case of Windows,
>>>> it would be the only real way for me to test that it's behaving exactly as
>>>> I want at this point.  This isn't the only time I've wanted C++-level tests
>>>> at a fairly fine granularity, but it's a good example of it.
>>>> > >
>>>> > > Any thoughts?
>>>> > > --
>>>> > > Todd Fiala |   Software Engineer |     tfiala at google.com |
>>>> 650-943-3180
>>>> > >
>>>> > > _______________________________________________
>>>> > > lldb-dev mailing list
>>>> > > lldb-dev at cs.uiuc.edu
>>>> > > http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
>>>> >
>>>> > _______________________________________________
>>>> > lldb-dev mailing list
>>>> > lldb-dev at cs.uiuc.edu
>>>> > http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
>>>> >
>>>> > _______________________________________________
>>>> > lldb-dev mailing list
>>>> > lldb-dev at cs.uiuc.edu
>>>> > http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
>>>>
>>>>
>>> _______________________________________________
>>> lldb-dev mailing list
>>> lldb-dev at cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
>>>
>>>
>>>
>>
>>
>
>
> --
> Todd Fiala | Software Engineer | tfiala at google.com
>
>


-- 
Todd Fiala | Software Engineer | tfiala at google.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20141002/fe8f5c2e/attachment.html>

More information about the lldb-dev mailing list