[lldb-dev] Is anyone using the gtest Xcode project?
Zachary Turner
zturner at google.com
Thu Mar 12 18:19:28 PDT 2015
That's interesting! So essentially we could use all of LLDB's existing test
infra *except* the part that builds the executables. That would at least
get rid of the need to write Makefiles.
I know we have some logic to compile the same executables multiple times
with different target architectures and even different compilers, but i
think lit supports this too because ASAN does something similar.
On Thu, Mar 12, 2015 at 6:05 PM Jonathan Roelofs <jonathan at codesourcery.com>
wrote:
>
>
> On 3/12/15 6:06 PM, Zachary Turner wrote:
> > Wasn't really trying to get into an extended discussion about this, but
> > FWIW I definitely realize that lldb's tests are more complicated than
> > what lit currently supports. But that's why I said "even if it meant
> > extending lit". It was mostly just a general comment about how it's
> > nice if everyone is focused on making one thing better instead of
> > everyone having different things.
> >
> > As for specifics, my understanding is that lit parallelizes better (so
> > running tests is faster), understands how to build programs (so doesn't
> > require makefiles), and has a richer language for specifying how and
> > under what circumstances different tests should be run. It's also
> > familiar to other LLVM developers (so encourages cross-collaboration),
> > and allows one to write self-contained tests with the program to test
> > and the check in a single file (less maintenance).
>
> I don't know much about lldb's testing needs, but I thought I'd chime in
> and mention that there's a lot of magic in the lit TestFormat (which
> testsuites can define themselves), and that it's waaaaay more flexible
> than it looks.
>
> LLVM uses the ShTest test format a lot, and that one is well-suited for
> the process Jim describes: building shell script pipelines with checkers
> at the end, where the test file itself serves as both the input to the
> beginning of the pipeline, and the input to the checker at the end.
>
> Libcxx and Libcxxabi have their own LibcxxTestFormat that behaves a
> little differently: they make the assumption that every test is a c/c++
> file which needs to be compiled, executed, and that an exit status of 0
> is a pass.
>
> From a quick look at the lldb testsuite, I imagine if you wanted to
> LIT-ify it, it would make sense to have yet another TestFormat. This
> test format would know that it needs to look for *.py files, invoke the
> Makefile next to them (or put metadata in comments at the top of the
> *.py that says how to build it), then run the *.py file. I think the
> changes to the tests themselves would be minimal, as would writing the
> lit.cfg to glue it all together.
>
>
> Cheers,
>
> Jon
> >
> > In any case, I'm really not an expert on lit, so +bogner and +chandlerc
> > in case they want to chime in. I do think it's at least worth thinking
> > about whether lit *could* be extended to meet LLDB's needs -- if nothing
> > else as a thought exercise, and maybe learning more about how it works
> > would give us some ideas to make our own test suite better.
> >
> >
> > On Thu, Mar 12, 2015 at 4:39 PM <jingham at apple.com
> > <mailto:jingham at apple.com>> wrote:
> >
> >
> > > On Mar 12, 2015, at 4:08 PM, Zachary Turner <zturner at google.com
> > <mailto:zturner at google.com>> wrote:
> > >
> > > Oh I'm all for reusing as much of the existing mechanism as
> > possible. Was just stating how the CMake worked as a discussion
> > point. Another possibility would be to just have the Xcode project
> > build one executable that pulls in sources recursively from the
> > entire subtree. Is this as easy in Xcode as just adding all sources
> > from a subfolder to a single target?
> >
> > >
> > > One day far off in the future it would be nice if all of LLDB's
> > tests were ported to lit (even if that meant extending lit to make
> > it do what we needed it to do),
> >
> > Why would this be nice? It looks like lit is a good test runner for
> > tests that have some input, do something with the input, produce an
> > output and check that output is matches some pattern. That is not
> > at all what the lldb tests look like. They often have to do complex
> > dances - for instance depending on how the line tables come out
> > there are many "correct" ways to step through code. If you are
> > going to test this you've got to do "step, if I got to a close
> > bracket, step again, if I got past it don't. Etc...
> >
> > I see no benefit in extending a simple runner like lit to do the
> > complex dances the lldb testsuite sometimes has to do. I'm all for
> > sharing, but it is also okay to have two implementations of some
> > functionality if the two uses are sufficiently different, and this
> > certainly seems like one of those cases.
> >
> > > so I can definitely see some value in hooking lit up to the Xcode
> > build so it does everything the CMake build does. I'll have to look
> > into exactly what steps the CMake and/or autoconf build are taking,
> > but I suspect it's going to involve running CMake from a script, so
> > not very desirable. I'm still learning a lot of this stuff though,
> > so there may be a better way. Either way, I'll have to look into it
> > a little bit.
> >
> >
> > Jim
> >
> >
> > >
> > > In the meantime, if running unit tests from Xcode is not part of
> > anyone's usual workflow, can I remove it for now?
> > >
> > > On Thu, Mar 12, 2015 at 4:01 PM <jingham at apple.com
> > <mailto:jingham at apple.com>> wrote:
> > > I'm not sure if this is what you meant, but I don't see a lot of
> > value in making an Xcode project that has targets for each of the
> > gtest binaries, and then tries to run the tests. Seems to me it
> > would be better if the gtest project just invokes whatever mechanism
> > the cmake build would do to run the tests. That's just another set
> > of things to keep in sync.
> > >
> > > It is sufficient to have a target that just does whatever steps
> > cmake/lit do to build the gtests & run them, if that is possible. I
> > guess if you can't do this without running cmake in the lldb
> > top-level directory that would be a problem. But it still seems
> > better to me to wire that up, than to have to add tests to both
> > Xcode & cmake.
> > >
> > > Jim
> > >
> > >
> > >
> > >
> > >
> > > > On Mar 12, 2015, at 3:46 PM, Zachary Turner <zturner at google.com
> > <mailto:zturner at google.com>> wrote:
> > > >
> > > > So I'm guessing the scheme runs do-gtest.py. I'd like to
> > delete that file as well as all the Makefiles in the directory if
> > possible. It seems like these files should be built using the
> > normal Xcode build system the same way the rest of LLDB is built.
> > > >
> > > > The way the CMake does it is that each test folder generates a
> > new executable. So right now it will build HostTests.exe,
> > ProcessLinuxTests.exe, and UtilityTests.exe. And then CMake will
> > invoke lit (the LLVM test runner) to run each of the executables one
> > by one and print the output.
> > > >
> > > > I'm not sure if that's easy or feasible to do in the Xcode
> > build. I kind of don't want to leave this do-gtest.py and Makefiles
> > in the build though, because the more of this stuff we have the more
> > maintenance it is, and things tend to rot.
> > > >
> > > >
> > > > On Thu, Mar 12, 2015 at 3:23 PM <jingham at apple.com
> > <mailto:jingham at apple.com>> wrote:
> > > > Xcode has "projects" and then "workspaces" and "schemes".
> > Workspaces aggregate projects. Schemes exist in both workspaces and
> > projects and are the way to say "do something with some of the stuff
> > referred to by this project/workspace." So the way to do this
> > formally is to have the gtest scheme build & run the tests from the
> > gtest project.
> > > >
> > > > The lldb.xcworkspace file does reference the gtest xcode
> > project, and it has a scheme for the gtest.
> > > >
> > > > Not sure what the scheme does yet, I'll look in a few minutes
> > if nobody beats me to it, I'm in the middle of things right now.
> > > >
> > > > Jim
> > > >
> > > >
> > > >
> > > > > On Mar 12, 2015, at 2:41 PM, Zachary Turner
> > <zturner at google.com <mailto:zturner at google.com>> wrote:
> > > > >
> > > > > In lldb/gtest there is a gtest.xcodeproj folder with what I
> > guess is an Xcode project. If I understand the way Xcode works, the
> > way to use this is by opening this in another instance of Xcode
> > separate from your normal LLDB project, and then building it. Is
> > this right?
> > > > >
> > > > > I have a patch that moves some files around, and if nobody is
> > using this Xcode project, I would like to delete it. Then, after I
> > get the tests up and running in the CMake build, we can add it to
> > the "real" Xcode project as a separate target similar to how you
> > currently run the LLDB Test suite.
> > > > >
> > > > > Any objections to deleting the Xcode project?
> > > >
> > >
> >
> >
> >
> > _______________________________________________
> > lldb-dev mailing list
> > lldb-dev at cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
> >
>
> --
> Jon Roelofs
> jonathan at codesourcery.com
> CodeSourcery / Mentor Embedded
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150313/8a6d07c6/attachment.html>
More information about the lldb-dev
mailing list