[lldb-dev] Is anyone using the gtest Xcode project?

[Zachary Turner]

Well, as a quick example of where I think there's a considerable amount of
overlap between the high level model of how the test operates is the case
of the lldbinline tests.

On Thu, Mar 12, 2015 at 5:28 PM <jingham at apple.com> wrote:

>
> > On Mar 12, 2015, at 5:06 PM, Zachary Turner <zturner at google.com> wrote:
> >
> > Wasn't really trying to get into an extended discussion about this, but
> FWIW I definitely realize that lldb's tests are more complicated than what
> lit currently supports.  But that's why I said "even if it meant extending
> lit".  It was mostly just a general comment about how it's nice if everyone
> is focused on making one thing better instead of everyone having different
> things.
> >
>
> Depending on how different the different things are.  Compiler tests tend
> to have input, output and some machine that converts the input to the
> output.  That is one very particular model of testing.  Debugger tests need
> to do: get to stage 1, if that succeeded, get to stage 2, if that
> succeeded, etc.  Plus there's generally substantial setup code to get
> somewhere interesting, so while you are there you generally try to test a
> bunch of similar things.  Plus, the tests often have points where there are
> several success cases, but each one requires a different "next action",
> stepping being the prime example of this.  These are very different models
> and I don't see that trying to smush the two together would be a fruitful
> exercise.
>
> Jim
>
> > As for specifics, my understanding is that lit parallelizes better (so
> running tests is faster), understands how to build programs (so doesn't
> require makefiles), and has a richer language for specifying how and under
> what circumstances different tests should be run.  It's also familiar to
> other LLVM developers (so encourages cross-collaboration), and allows one
> to write self-contained tests with the program to test and the check in a
> single file (less maintenance).
> >
> > In any case, I'm really not an expert on lit, so +bogner and +chandlerc
> in case they want to chime in.  I do think it's at least worth thinking
> about whether lit *could* be extended to meet LLDB's needs -- if nothing
> else as a thought exercise, and maybe learning more about how it works
> would give us some ideas to make our own test suite better.
> >
> >
> > On Thu, Mar 12, 2015 at 4:39 PM <jingham at apple.com> wrote:
> >
> > > On Mar 12, 2015, at 4:08 PM, Zachary Turner <zturner at google.com>
> wrote:
> > >
> > > Oh I'm all for reusing as much of the existing mechanism as possible.
> Was just stating how the CMake worked as a discussion point.  Another
> possibility would be to just have the Xcode project build one executable
> that pulls in sources recursively from the entire subtree.  Is this as easy
> in Xcode as just adding all sources from a subfolder to a single target?
> >
> > >
> > > One day far off in the future it would be nice if all of LLDB's tests
> were ported to lit (even if that meant extending lit to make it do what we
> needed it to do),
> >
> > Why would this be nice?  It looks like lit is a good test runner for
> tests that have some input, do something with the input, produce an output
> and check that output is matches some pattern.  That is not at all what the
> lldb tests look like.  They often have to do complex dances - for instance
> depending on how the line tables come out there are many "correct" ways to
> step through code.  If you are going to test this you've got to do "step,
> if I got to a close bracket, step again, if I got past it don't.  Etc...
> >
> > I see no benefit in extending a simple runner like lit to do the complex
> dances the lldb testsuite sometimes has to do.  I'm all for sharing, but it
> is also okay to have two implementations of some functionality if the two
> uses are sufficiently different, and this certainly seems like one of those
> cases.
> >
> > > so I can definitely see some value in hooking lit up to the Xcode
> build so it does everything the CMake build does.  I'll have to look into
> exactly what steps the CMake and/or autoconf build are taking, but I
> suspect it's going to involve running CMake from a script, so not very
> desirable.  I'm still learning a lot of this stuff though, so there may be
> a better way.  Either way, I'll have to look into it a little bit.
> >
> >
> > Jim
> >
> >
> > >
> > > In the meantime, if running unit tests from Xcode is not part of
> anyone's usual workflow, can I remove it for now?
> > >
> > > On Thu, Mar 12, 2015 at 4:01 PM <jingham at apple.com> wrote:
> > > I'm not sure if this is what you meant, but I don't see a lot of value
> in making an Xcode project that has targets for each of the gtest binaries,
> and then tries to run the tests.  Seems to me it would be better if the
> gtest project just invokes whatever mechanism the cmake build would do to
> run the tests.  That's just another set of things to keep in sync.
> > >
> > > It is sufficient to have a target that just does whatever steps
> cmake/lit do to build the gtests & run them, if that is possible.  I guess
> if you can't do this without running cmake in the lldb top-level directory
> that would be a problem.  But it still seems better to me to wire that up,
> than to have to add tests to both Xcode & cmake.
> > >
> > > Jim
> > >
> > >
> > >
> > >
> > >
> > > > On Mar 12, 2015, at 3:46 PM, Zachary Turner <zturner at google.com>
> wrote:
> > > >
> > > > So I'm guessing the scheme runs do-gtest.py.  I'd like to delete
> that file as well as all the Makefiles in the directory if possible.  It
> seems like these files should be built using the normal Xcode build system
> the same way the rest of LLDB is built.
> > > >
> > > > The way the CMake does it is that each test folder generates a new
> executable.  So right now it will build HostTests.exe,
> ProcessLinuxTests.exe, and UtilityTests.exe.  And then CMake will invoke
> lit (the LLVM test runner) to run each of the executables one by one and
> print the output.
> > > >
> > > > I'm not sure if that's easy or feasible to do in the Xcode build.  I
> kind of don't want to leave this do-gtest.py and Makefiles in the build
> though, because the more of this stuff we have the more maintenance it is,
> and things tend to rot.
> > > >
> > > >
> > > > On Thu, Mar 12, 2015 at 3:23 PM <jingham at apple.com> wrote:
> > > > Xcode has "projects" and then "workspaces" and "schemes".
> Workspaces aggregate projects.  Schemes exist in both workspaces and
> projects and are the way to say "do something with some of the stuff
> referred to by this project/workspace."  So the way to do this formally is
> to have the gtest scheme build & run the tests from the gtest project.
> > > >
> > > > The lldb.xcworkspace file does reference the gtest xcode project,
> and it has a scheme for the gtest.
> > > >
> > > > Not sure what the scheme does yet, I'll look in a few minutes if
> nobody beats me to it, I'm in the middle of things right now.
> > > >
> > > > Jim
> > > >
> > > >
> > > >
> > > > > On Mar 12, 2015, at 2:41 PM, Zachary Turner <zturner at google.com>
> wrote:
> > > > >
> > > > > In lldb/gtest there is a gtest.xcodeproj folder with what I guess
> is an Xcode project.  If I understand the way Xcode works, the way to use
> this is by opening this in another instance of Xcode separate from your
> normal LLDB project, and then building it.  Is this right?
> > > > >
> > > > > I have a patch that moves some files around, and if nobody is
> using this Xcode project, I would like to delete it.  Then, after I get the
> tests up and running in the CMake build, we can add it to the "real" Xcode
> project as a separate target similar to how you currently run the LLDB Test
> suite.
> > > > >
> > > > > Any objections to deleting the Xcode project?
> > > >
> > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150313/1cd18ed8/attachment.html>