That's interesting! So essentially we could use all of LLDB's existing test infra *except* the part that builds the executables. That would at least get rid of the need to write Makefiles. <br><br>I know we have some logic to compile the same executables multiple times with different target architectures and even different compilers, but i think lit supports this too because ASAN does something similar.<br><div class="gmail_quote">On Thu, Mar 12, 2015 at 6:05 PM Jonathan Roelofs <<a href="mailto:jonathan@codesourcery.com">jonathan@codesourcery.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
<br>
On 3/12/15 6:06 PM, Zachary Turner wrote:<br>
> Wasn't really trying to get into an extended discussion about this, but<br>
> FWIW I definitely realize that lldb's tests are more complicated than<br>
> what lit currently supports. But that's why I said "even if it meant<br>
> extending lit". It was mostly just a general comment about how it's<br>
> nice if everyone is focused on making one thing better instead of<br>
> everyone having different things.<br>
><br>
> As for specifics, my understanding is that lit parallelizes better (so<br>
> running tests is faster), understands how to build programs (so doesn't<br>
> require makefiles), and has a richer language for specifying how and<br>
> under what circumstances different tests should be run. It's also<br>
> familiar to other LLVM developers (so encourages cross-collaboration),<br>
> and allows one to write self-contained tests with the program to test<br>
> and the check in a single file (less maintenance).<br>
<br>
I don't know much about lldb's testing needs, but I thought I'd chime in<br>
and mention that there's a lot of magic in the lit TestFormat (which<br>
testsuites can define themselves), and that it's waaaaay more flexible<br>
than it looks.<br>
<br>
LLVM uses the ShTest test format a lot, and that one is well-suited for<br>
the process Jim describes: building shell script pipelines with checkers<br>
at the end, where the test file itself serves as both the input to the<br>
beginning of the pipeline, and the input to the checker at the end.<br>
<br>
Libcxx and Libcxxabi have their own LibcxxTestFormat that behaves a<br>
little differently: they make the assumption that every test is a c/c++<br>
file which needs to be compiled, executed, and that an exit status of 0<br>
is a pass.<br>
<br>
From a quick look at the lldb testsuite, I imagine if you wanted to<br>
LIT-ify it, it would make sense to have yet another TestFormat. This<br>
test format would know that it needs to look for *.py files, invoke the<br>
Makefile next to them (or put metadata in comments at the top of the<br>
*.py that says how to build it), then run the *.py file. I think the<br>
changes to the tests themselves would be minimal, as would writing the<br>
lit.cfg to glue it all together.<br>
<br>
<br>
Cheers,<br>
<br>
Jon<br>
><br>
> In any case, I'm really not an expert on lit, so +bogner and +chandlerc<br>
> in case they want to chime in. I do think it's at least worth thinking<br>
> about whether lit *could* be extended to meet LLDB's needs -- if nothing<br>
> else as a thought exercise, and maybe learning more about how it works<br>
> would give us some ideas to make our own test suite better.<br>
><br>
><br>
> On Thu, Mar 12, 2015 at 4:39 PM <<a href="mailto:jingham@apple.com" target="_blank">jingham@apple.com</a><br>
> <mailto:<a href="mailto:jingham@apple.com" target="_blank">jingham@apple.com</a>>> wrote:<br>
><br>
><br>
> > On Mar 12, 2015, at 4:08 PM, Zachary Turner <<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a><br>
> <mailto:<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>>> wrote:<br>
> ><br>
> > Oh I'm all for reusing as much of the existing mechanism as<br>
> possible. Was just stating how the CMake worked as a discussion<br>
> point. Another possibility would be to just have the Xcode project<br>
> build one executable that pulls in sources recursively from the<br>
> entire subtree. Is this as easy in Xcode as just adding all sources<br>
> from a subfolder to a single target?<br>
><br>
> ><br>
> > One day far off in the future it would be nice if all of LLDB's<br>
> tests were ported to lit (even if that meant extending lit to make<br>
> it do what we needed it to do),<br>
><br>
> Why would this be nice? It looks like lit is a good test runner for<br>
> tests that have some input, do something with the input, produce an<br>
> output and check that output is matches some pattern. That is not<br>
> at all what the lldb tests look like. They often have to do complex<br>
> dances - for instance depending on how the line tables come out<br>
> there are many "correct" ways to step through code. If you are<br>
> going to test this you've got to do "step, if I got to a close<br>
> bracket, step again, if I got past it don't. Etc...<br>
><br>
> I see no benefit in extending a simple runner like lit to do the<br>
> complex dances the lldb testsuite sometimes has to do. I'm all for<br>
> sharing, but it is also okay to have two implementations of some<br>
> functionality if the two uses are sufficiently different, and this<br>
> certainly seems like one of those cases.<br>
><br>
> > so I can definitely see some value in hooking lit up to the Xcode<br>
> build so it does everything the CMake build does. I'll have to look<br>
> into exactly what steps the CMake and/or autoconf build are taking,<br>
> but I suspect it's going to involve running CMake from a script, so<br>
> not very desirable. I'm still learning a lot of this stuff though,<br>
> so there may be a better way. Either way, I'll have to look into it<br>
> a little bit.<br>
><br>
><br>
> Jim<br>
><br>
><br>
> ><br>
> > In the meantime, if running unit tests from Xcode is not part of<br>
> anyone's usual workflow, can I remove it for now?<br>
> ><br>
> > On Thu, Mar 12, 2015 at 4:01 PM <<a href="mailto:jingham@apple.com" target="_blank">jingham@apple.com</a><br>
> <mailto:<a href="mailto:jingham@apple.com" target="_blank">jingham@apple.com</a>>> wrote:<br>
> > I'm not sure if this is what you meant, but I don't see a lot of<br>
> value in making an Xcode project that has targets for each of the<br>
> gtest binaries, and then tries to run the tests. Seems to me it<br>
> would be better if the gtest project just invokes whatever mechanism<br>
> the cmake build would do to run the tests. That's just another set<br>
> of things to keep in sync.<br>
> ><br>
> > It is sufficient to have a target that just does whatever steps<br>
> cmake/lit do to build the gtests & run them, if that is possible. I<br>
> guess if you can't do this without running cmake in the lldb<br>
> top-level directory that would be a problem. But it still seems<br>
> better to me to wire that up, than to have to add tests to both<br>
> Xcode & cmake.<br>
> ><br>
> > Jim<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > > On Mar 12, 2015, at 3:46 PM, Zachary Turner <<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a><br>
> <mailto:<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>>> wrote:<br>
> > ><br>
> > > So I'm guessing the scheme runs do-gtest.py. I'd like to<br>
> delete that file as well as all the Makefiles in the directory if<br>
> possible. It seems like these files should be built using the<br>
> normal Xcode build system the same way the rest of LLDB is built.<br>
> > ><br>
> > > The way the CMake does it is that each test folder generates a<br>
> new executable. So right now it will build HostTests.exe,<br>
> ProcessLinuxTests.exe, and UtilityTests.exe. And then CMake will<br>
> invoke lit (the LLVM test runner) to run each of the executables one<br>
> by one and print the output.<br>
> > ><br>
> > > I'm not sure if that's easy or feasible to do in the Xcode<br>
> build. I kind of don't want to leave this do-gtest.py and Makefiles<br>
> in the build though, because the more of this stuff we have the more<br>
> maintenance it is, and things tend to rot.<br>
> > ><br>
> > ><br>
> > > On Thu, Mar 12, 2015 at 3:23 PM <<a href="mailto:jingham@apple.com" target="_blank">jingham@apple.com</a><br>
> <mailto:<a href="mailto:jingham@apple.com" target="_blank">jingham@apple.com</a>>> wrote:<br>
> > > Xcode has "projects" and then "workspaces" and "schemes".<br>
> Workspaces aggregate projects. Schemes exist in both workspaces and<br>
> projects and are the way to say "do something with some of the stuff<br>
> referred to by this project/workspace." So the way to do this<br>
> formally is to have the gtest scheme build & run the tests from the<br>
> gtest project.<br>
> > ><br>
> > > The lldb.xcworkspace file does reference the gtest xcode<br>
> project, and it has a scheme for the gtest.<br>
> > ><br>
> > > Not sure what the scheme does yet, I'll look in a few minutes<br>
> if nobody beats me to it, I'm in the middle of things right now.<br>
> > ><br>
> > > Jim<br>
> > ><br>
> > ><br>
> > ><br>
> > > > On Mar 12, 2015, at 2:41 PM, Zachary Turner<br>
> <<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a> <mailto:<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>>> wrote:<br>
> > > ><br>
> > > > In lldb/gtest there is a gtest.xcodeproj folder with what I<br>
> guess is an Xcode project. If I understand the way Xcode works, the<br>
> way to use this is by opening this in another instance of Xcode<br>
> separate from your normal LLDB project, and then building it. Is<br>
> this right?<br>
> > > ><br>
> > > > I have a patch that moves some files around, and if nobody is<br>
> using this Xcode project, I would like to delete it. Then, after I<br>
> get the tests up and running in the CMake build, we can add it to<br>
> the "real" Xcode project as a separate target similar to how you<br>
> currently run the LLDB Test suite.<br>
> > > ><br>
> > > > Any objections to deleting the Xcode project?<br>
> > ><br>
> ><br>
><br>
><br>
><br>
> ______________________________<u></u>_________________<br>
> lldb-dev mailing list<br>
> <a href="mailto:lldb-dev@cs.uiuc.edu" target="_blank">lldb-dev@cs.uiuc.edu</a><br>
> <a href="http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev" target="_blank">http://lists.cs.uiuc.edu/<u></u>mailman/listinfo/lldb-dev</a><br>
><br>
<br>
--<br>
Jon Roelofs<br>
<a href="mailto:jonathan@codesourcery.com" target="_blank">jonathan@codesourcery.com</a><br>
CodeSourcery / Mentor Embedded<br>
</blockquote></div>