[lldb-dev] Use of MAKE in tests

[jingham at apple.com]

> On Jul 20, 2014, at 1:31 PM, Zachary Turner <zturner at google.com> wrote:
> 
> I have a couple of ideas here.  But before I list them I wanted to ask how you guys build LLVM.  LLVM doesn't have a native Xcode build, so presumably you're already running CMake once in order to build LLVM for LLDB to link against.  Can you expand on this workflow a little?  
> 

There is a script in the scripts folder (build-llvm.pl) that gets invoked from the Xcode build of lldb.  That checks out (if necessary) and builds llvm/clang.  That script used to use the Makefile build but we switched that to cmake a while ago.  That script is invoked by one of the lldb Xcode targets.  So from Xcode you just build the overall lldb-tool target and that builds all the needed bits including fetching and building llvm/clang if necessary.  BTW in our case, if necessary means if the llvm product .a file is absent since most of us don't change clang and it is annoying to have the llvm make figure out if it needs to do anything every time you change an lldb file...


> Anyway, here's some options.  Maybe there's more, this just what I came up with after a little brainstorming
> 
> 1) Building the test suite on MacOSX requires an additional step beyond just building LLDB.  You would have to build the test programs first, presumably the same way that you build LLVM assuming you use the CMake llvm build, then the test script would find the already-built test programs.  Note that this is the same as what non MacOSX people would do, except that for us it wouldn't be an extra step, simply typing "ninja check-lldb" would automatically do everything, since building the test programs would be a dependency of running the test suite.
> 

As Todd pointed out, there are some tests that rely on building and then hiding files, or building the tests in different ways, or stripping some of the output, etc.  So I think the model where you build all the test programs up front isn't a great one for our test suite.  If you wanted to use cmake, it would make more sense to have a pass that builds the Makefiles from the cmake whatever their input files are called, and then the actual test suite run would build the needed binaries using the Makefiles pretty much as it did before.  Or if cmake can be used directly - like a Makefile - then we could switch the code in the testsuite to use them directly instead of invoking make.  But anyway, "making the makefiles" can be done outside the test harness, but building the binaries needed for the tests still needs to be controlled by the test code directly.

> 2) Have parallel builds for the test programs on MacOSX and other platforms.  MacOSX build would remain as-is, and other platforms would use CMake to build the tests.  Maybe there could be an option in the test script to force MAKE-mode for backwards compatibility.  This isn't that different from the current situation, since MacOSX and other platforms already have parallel build systems for LLDB itself.  The biggest difference is that now when adding new tests, you would have to add both a CMake file and a Makefile.  This is a small annoyance, but looking at the makefiles, they are all pretty simple and most of the logic is factored into the top-level Makefile.  I think the same could be done for the CMake, so that new CMake files are a few lines.

This seems awkward.

> 
> 3) Have MacOSX not use the native Xcode build anymore.  This goes back to my original question though, about what the workflow is like.  I imagine it's probably just historical, but why doesn't LLDB use the CMake build already?  CMake can already generate Xcode projects, what are the differences between the Xcode projects generated by CMake versus the hand-maintained Xcode projects.  Assuming they're significant enough to warrant serious concerns about the usability of a CMake-generated Xcode project, I wonder if it's something that's fixable to the point that using a CMake-generated Xcode project is almost the same as using the hand-maintained Xcode project.  I've never used Xcode before, so I'm a bit clueless here.

We (or at least some of us) use Xcode because we like it.  Dunno how easy it would be to get cmake to construct well-organized Xcode projects.  Before we embark down this path - given that some of us really do use Xcode as our development platform - we'd have to have not just confidence not just that whatever difficulties there are in constructing useable projects could be overcome, but that there's somebody who uses Xcode who wants to maintain this bit.  Having spent many years maintaining complex auto-conf based projects, I must admit I personally am very happy in my ignorance of cmake.

Jim

> 
> 
> On Fri, Jul 18, 2014 at 6:20 PM, <jingham at apple.com> wrote:
> That won't work on the Mac for Xcode builds, which is what pretty much all of us here use to build lldb.
> 
> Jim
> 
> > On Jul 18, 2014, at 6:03 PM, Zachary Turner <zturner at google.com> wrote:
> >
> > I don't think it's that simple, because some of the test have to exercise features of the debugger that deal with how your program was built.  Was it built with symbols, without symbols, with FPO, are the symbols in a different folder than the exe (e.g. testing the symbol search path), etc.
> >
> > I thought about it a little, because I agree with you that it's annoying to have the test run CMake and then depend on some external build tool, but I think an elegant way to make this work is to just have all of the test executables and libraries built at the same time you build LLDB.  The nice thing about this is that the tests end up running significantly faster, because simply running the test suite will never end up building anything, it will just load executables that have already been built.  It also avoids the additional dependency problem, because you're already building LLDB with CMake to begin with.
> >
> >
> > On Fri, Jul 18, 2014 at 5:24 PM, Reid Kleckner <rnk at google.com> wrote:
> > I haven't stared deeply at LLDB's usage of make, but I think adding CMake to the mix is a step in the wrong direction.  CMake isn't a make replacement, it's a meta-build system and autoconf replacement.  Once you generate build files, you'll still have an external dependency on some other build system (ninja or msbuild).
> >
> > Aren't most LLDB test programs simple enough to be linked with something like the lit shell interpreter?
> >
> >
> > On Fri, Jul 18, 2014 at 3:55 PM, <jingham at apple.com> wrote:
> > This is mostly historical.  So long as I don't have to do it, I don't really mind one way or the other.  OTOH, beware, this sort of thing is sticky...
> >
> > Jim
> >
> >
> > > On Jul 18, 2014, at 3:47 PM, Zachary Turner <zturner at google.com> wrote:
> > >
> > > I'm not very well versed in MAKE, but an issue I'm encountering currently is the use of MAKE in compiling the test files.  Is there any reason CMake can't be used here?  There's a lot of shell-specific logic in the Makefile currently that is hard to reproduce on Windows, so if we can port everything to CMake it might be helpful.
> > >
> > > Thoughts?
> > > _______________________________________________
> > > lldb-dev mailing list
> > > lldb-dev at cs.uiuc.edu
> > > http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
> >
> > _______________________________________________
> > lldb-dev mailing list
> > lldb-dev at cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
> >
> >
> 
>