[lldb-dev] Is anyone using the gtest Xcode project?

[Zachary Turner]

Also, after the dust settles I will then go and add a gtest scheme back to
the xcode workspace as we discussed previously (again, unless someone beats
me to it)

On Fri, Mar 13, 2015 at 1:26 PM Zachary Turner <zturner at google.com> wrote:

> I got confirmation from Vince offline that we don't need gtest in the
> Xcode workspace in its current form (i.e. the scheme that runs the
> do-gtest.py).  So I'm going to check in my changes which add gtest to the
> CMake and delete this xcodeproj from the repo.  This may result in errors
> in the Xcode workspace the next time you load it up.  This should be as
> easy to fix as removing the reference to gtest.xcodeproj.
>
> I will try to figure out how to do that later today as well if nobody
> beats me to it, but I have a few things I need to get to first.
>
>
> On Thu, Mar 12, 2015 at 6:19 PM <jingham at apple.com> wrote:
>
>> Right, you could either do this in the lldb.xcodeproj or in a separate
>> project which you include in the lldb.xcworkspace.  There's no real reason
>> to cram everything into the lldb.xcodeproj, as long as it is added to the
>> workspace it is easy to access it, set up dependencies, etc.
>>
>> Jim
>>
>> > On Mar 12, 2015, at 6:14 PM, Zachary Turner <zturner at google.com> wrote:
>> >
>> > Ahh yea, i think it's fine the way you describe (explicitly adding each
>> one to the right target), because that's how the Xcode project works for
>> regular LLDB no? So we could have a Tests folder in Xcode, which contains
>> Host, Plugins, and Utility folders, and those folders contain more files
>> (or subfolders) and all of this gets compiled into a single executable
>> named lldb-unit-tests.
>> > On Thu, Mar 12, 2015 at 6:08 PM <jingham at apple.com> wrote:
>> >
>> > > On Mar 12, 2015, at 6:00 PM, Zachary Turner <zturner at google.com>
>> wrote:
>> > >
>> > > Well, like I said.  I'm just thinking :)  No need to worry
>> > >
>> > > Back to the original question, is it as easy as it seems to just
>> create one target in Xcode that manually includes each file recursively in
>> the subtree?  So it just builds one executable?
>> > >
>> >
>> > I don't think so. You can ADD a folder of sources to a project, but
>> that just makes all the files available to the project.  You then have to
>> manually tell Xcode which files build into which targets.  That's pretty
>> easy to do, but I don't know of a way to  get it to "include all .c files
>> in the current target."
>> >
>> > Jim
>> >
>> > (I removed Chandler & Justin 'cause I doubt they care about Xcode...)
>> >
>> >
>> > > On Thu, Mar 12, 2015 at 5:56 PM <jingham at apple.com> wrote:
>> > > The lldbinline  tests are an okay way to write a very simple class of
>> tests.  But they will not suffice for many of the tests we need to write.
>> I am actually not a big fan of these tests because when they fail it is a
>> royal pain to reproduce the steps that led to the failure.  I don't think
>> making a wholly different runner to run this is going to make that
>> situation any better.
>> > >
>> > > Jim
>> > >
>> > >
>> > > > On Mar 12, 2015, at 5:38 PM, Zachary Turner <zturner at google.com>
>> wrote:
>> > > >
>> > > > Well, as a quick example of where I think there's a considerable
>> amount of overlap between the high level model of how the test operates is
>> the case of the lldbinline tests.
>> > > >
>> > > > On Thu, Mar 12, 2015 at 5:28 PM <jingham at apple.com> wrote:
>> > > >
>> > > > > On Mar 12, 2015, at 5:06 PM, Zachary Turner <zturner at google.com>
>> wrote:
>> > > > >
>> > > > > Wasn't really trying to get into an extended discussion about
>> this, but FWIW I definitely realize that lldb's tests are more complicated
>> than what lit currently supports.  But that's why I said "even if it meant
>> extending lit".  It was mostly just a general comment about how it's nice
>> if everyone is focused on making one thing better instead of everyone
>> having different things.
>> > > > >
>> > > >
>> > > > Depending on how different the different things are.  Compiler
>> tests tend to have input, output and some machine that converts the input
>> to the output.  That is one very particular model of testing.  Debugger
>> tests need to do: get to stage 1, if that succeeded, get to stage 2, if
>> that succeeded, etc.  Plus there's generally substantial setup code to get
>> somewhere interesting, so while you are there you generally try to test a
>> bunch of similar things.  Plus, the tests often have points where there are
>> several success cases, but each one requires a different "next action",
>> stepping being the prime example of this.  These are very different models
>> and I don't see that trying to smush the two together would be a fruitful
>> exercise.
>> > > >
>> > > > Jim
>> > > >
>> > > > > As for specifics, my understanding is that lit parallelizes
>> better (so running tests is faster), understands how to build programs (so
>> doesn't require makefiles), and has a richer language for specifying how
>> and under what circumstances different tests should be run.  It's also
>> familiar to other LLVM developers (so encourages cross-collaboration), and
>> allows one to write self-contained tests with the program to test and the
>> check in a single file (less maintenance).
>> > > > >
>> > > > > In any case, I'm really not an expert on lit, so +bogner and
>> +chandlerc in case they want to chime in.  I do think it's at least worth
>> thinking about whether lit *could* be extended to meet LLDB's needs -- if
>> nothing else as a thought exercise, and maybe learning more about how it
>> works would give us some ideas to make our own test suite better.
>> > > > >
>> > > > >
>> > > > > On Thu, Mar 12, 2015 at 4:39 PM <jingham at apple.com> wrote:
>> > > > >
>> > > > > > On Mar 12, 2015, at 4:08 PM, Zachary Turner <zturner at google.com>
>> wrote:
>> > > > > >
>> > > > > > Oh I'm all for reusing as much of the existing mechanism as
>> possible.  Was just stating how the CMake worked as a discussion point.
>> Another possibility would be to just have the Xcode project build one
>> executable that pulls in sources recursively from the entire subtree.  Is
>> this as easy in Xcode as just adding all sources from a subfolder to a
>> single target?
>> > > > >
>> > > > > >
>> > > > > > One day far off in the future it would be nice if all of LLDB's
>> tests were ported to lit (even if that meant extending lit to make it do
>> what we needed it to do),
>> > > > >
>> > > > > Why would this be nice?  It looks like lit is a good test runner
>> for tests that have some input, do something with the input, produce an
>> output and check that output is matches some pattern.  That is not at all
>> what the lldb tests look like.  They often have to do complex dances - for
>> instance depending on how the line tables come out there are many "correct"
>> ways to step through code.  If you are going to test this you've got to do
>> "step, if I got to a close bracket, step again, if I got past it don't.
>> Etc...
>> > > > >
>> > > > > I see no benefit in extending a simple runner like lit to do the
>> complex dances the lldb testsuite sometimes has to do.  I'm all for
>> sharing, but it is also okay to have two implementations of some
>> functionality if the two uses are sufficiently different, and this
>> certainly seems like one of those cases.
>> > > > >
>> > > > > > so I can definitely see some value in hooking lit up to the
>> Xcode build so it does everything the CMake build does.  I'll have to look
>> into exactly what steps the CMake and/or autoconf build are taking, but I
>> suspect it's going to involve running CMake from a script, so not very
>> desirable.  I'm still learning a lot of this stuff though, so there may be
>> a better way.  Either way, I'll have to look into it a little bit.
>> > > > >
>> > > > >
>> > > > > Jim
>> > > > >
>> > > > >
>> > > > > >
>> > > > > > In the meantime, if running unit tests from Xcode is not part
>> of anyone's usual workflow, can I remove it for now?
>> > > > > >
>> > > > > > On Thu, Mar 12, 2015 at 4:01 PM <jingham at apple.com> wrote:
>> > > > > > I'm not sure if this is what you meant, but I don't see a lot
>> of value in making an Xcode project that has targets for each of the gtest
>> binaries, and then tries to run the tests.  Seems to me it would be better
>> if the gtest project just invokes whatever mechanism the cmake build would
>> do to run the tests.  That's just another set of things to keep in sync.
>> > > > > >
>> > > > > > It is sufficient to have a target that just does whatever steps
>> cmake/lit do to build the gtests & run them, if that is possible.  I guess
>> if you can't do this without running cmake in the lldb top-level directory
>> that would be a problem.  But it still seems better to me to wire that up,
>> than to have to add tests to both Xcode & cmake.
>> > > > > >
>> > > > > > Jim
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > > On Mar 12, 2015, at 3:46 PM, Zachary Turner <
>> zturner at google.com> wrote:
>> > > > > > >
>> > > > > > > So I'm guessing the scheme runs do-gtest.py.  I'd like to
>> delete that file as well as all the Makefiles in the directory if
>> possible.  It seems like these files should be built using the normal Xcode
>> build system the same way the rest of LLDB is built.
>> > > > > > >
>> > > > > > > The way the CMake does it is that each test folder generates
>> a new executable.  So right now it will build HostTests.exe,
>> ProcessLinuxTests.exe, and UtilityTests.exe.  And then CMake will invoke
>> lit (the LLVM test runner) to run each of the executables one by one and
>> print the output.
>> > > > > > >
>> > > > > > > I'm not sure if that's easy or feasible to do in the Xcode
>> build.  I kind of don't want to leave this do-gtest.py and Makefiles in the
>> build though, because the more of this stuff we have the more maintenance
>> it is, and things tend to rot.
>> > > > > > >
>> > > > > > >
>> > > > > > > On Thu, Mar 12, 2015 at 3:23 PM <jingham at apple.com> wrote:
>> > > > > > > Xcode has "projects" and then "workspaces" and "schemes".
>> Workspaces aggregate projects.  Schemes exist in both workspaces and
>> projects and are the way to say "do something with some of the stuff
>> referred to by this project/workspace."  So the way to do this formally is
>> to have the gtest scheme build & run the tests from the gtest project.
>> > > > > > >
>> > > > > > > The lldb.xcworkspace file does reference the gtest xcode
>> project, and it has a scheme for the gtest.
>> > > > > > >
>> > > > > > > Not sure what the scheme does yet, I'll look in a few minutes
>> if nobody beats me to it, I'm in the middle of things right now.
>> > > > > > >
>> > > > > > > Jim
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > > On Mar 12, 2015, at 2:41 PM, Zachary Turner <
>> zturner at google.com> wrote:
>> > > > > > > >
>> > > > > > > > In lldb/gtest there is a gtest.xcodeproj folder with what I
>> guess is an Xcode project.  If I understand the way Xcode works, the way to
>> use this is by opening this in another instance of Xcode separate from your
>> normal LLDB project, and then building it.  Is this right?
>> > > > > > > >
>> > > > > > > > I have a patch that moves some files around, and if nobody
>> is using this Xcode project, I would like to delete it.  Then, after I get
>> the tests up and running in the CMake build, we can add it to the "real"
>> Xcode project as a separate target similar to how you currently run the
>> LLDB Test suite.
>> > > > > > > >
>> > > > > > > > Any objections to deleting the Xcode project?
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150313/6888df2d/attachment.html>