[LLVMdev] RFC: Upcoming Build System Changes

[Marc J. Driftmeyer]

Daniel,

I'm just a 42 year old former NeXT and Apple Dev who back in Mechanical 
Engineering is excited about OpenCL/OpenGL and all that is with 
LLVM/Clang, but I've got to say that even though my plans include 
learning Python for various areas of development the last I want to deal 
with is a Build system needing it to compile a Compiler Suite.

Autotools is a bag of hurt, always has been.

Cmake improves considerably on it with it's CmakeCache.txt giving a 
cleanly prepared configuration one expects to see their project follow 
and doesn't drown one in a list of ten thousand lines of crap just to 
build something as common as Gimp, Inkscape, GTK+, GNUstep, Poppler, 
Freetype, POVRay, Bullet Library, Blender and any other project that 
also has their own internal build stages whether it's SCons or Python.

Why in the hell should one need to become well-versed in another 
language like Python just to customize how Cmake can build a project, 
when fixing the documentation and explanation of Cmake seems a more 
astute solution?

I've read that Cmake is a dog on Xcode. Fix support for it in Xcode.

Autotools and make for Project Builder/IB/EOF/WOF was garbage and had 
tons of custom hacks at NeXT and Apple. They didn't abandon it. They 
improved upon it to suite the needs of our products.

Then this complaint about build times and extra CPU cycles when you're 
living in a world of systems soon to average 16GB of RAM, 4-12 cores and 
GPUs that would make any old Animator dream back in the '90s really 
makes me laugh.

Reinventing the wheel 50 ways instead of improving the wheel until it's 
consistent, clean and easily extendable by third parties seems to be a 
worthy endeavor.

I love my Apple stock and all the hard work folks have put in to making 
great projects under Steve's vision, but to read about how Cmake is in 
bad shape with Xcode is something I cannot imagine Tevanian, Enderby, 
Ozer or other Engineers would have ever thought were such a daunting 
task that it cannot be resolved.

With all the brilliant minds on this list, you'd think they'd be able to 
coordinate a Proposal to improve Cmake upstream and provide a solution 
all parties could love.

You all get paid exceedingly well in an industry we all work in and to 
read about having to maintain a system that isn't perfect won't garner 
any compassion from me, or any reasonable person who knows 90% of life 
is a grind.

I would think these minds could convince the Cmake devs to improve their 
system and make it scalable as Lattner envisions without having to add 
Python or any other third party language into the mix and ultimately 
pair back down to Cmake.

Fix the engine. Worry about the suspension, handling, lights, sound 
system and the rest of the fluff later.

I read a lot of conjecture, hypotheticals and opinion.

Oscar seems to recognize the Keep It Simple Stupid mantra that any first 
year Mechanical Engineering student learns studying Machine Design so I 
would implore you, Oscar, to lead a Documentatio Project on Cmake with 
LLVM/Clang so us ``Jane `LLVM Developer' Doe'' types cannot only digest 
an abstract but also drill down into the manual with clear and concise 
examples--something I've always found lacking in the majority of the Man 
System legacy of UNIX.

I used to write NeXTAnswer solutions and support Enterprise Clients who 
could give a rat's behind about learning a bazillion little wrinkles on 
getting a solution to their trouble ticket(s).

Working in SQA at NeXT taught me to wear many hats. Relevant 
documentation cross-reference by revisions of a project where solutions 
worked is also a must. [Works under llvm-2.8, llvm-2.9, clang-2.7, 
clang-2.8, etc]

Like I said, my voice doesn't mean squat, but I'd suggest before trying 
to put lipstick on a pig, you have the solution completed, tested and 
fully documented. It goes much farther in convincing people to learn it.

By the way, the default Python version on Debian [and it's many spawned 
Distros] is 2.7. On OS X I've got 2.7-3.2. I've got the same on Linux.

On both I have only the current Cmake.

Reading through the thread tells me there is a lot of brainstorming to 
do and a lot of issues that are triggered, depending on how the 
implementation gets decided.

However it falls out, I'd hope a fully documented, living system of 
documentation is produced so no one has to waste the time of most senior 
developers on this list on basic ways of building, testing and deploying 
projects build against LLVM/Clang, whether it's on OS X, Linux, FreeBSD, 
Windows, Solaris, or whatever other platform supports LLVM/Clang.

It seems to me that part of being an architect [not my job] and 
implementing such a vast collection of projects under the LLVM/Clang 
umbrella requires those same folks to be fluent Technical Writers who 
can drill down to the average lamen.

- Marc J. Driftmeyer


On 10/27/2011 06:11 PM, Daniel Dunbar wrote:
> Hi all,
>
> As you might have inferred, I'm in the process of working on some changes to the
> way LLVM builds. I have outlined a rough proposal below, unless there are any
> major objections I will probably start committing stuff next week.
>
> This message may be verbose, if you want the executive summary, skip
> to 'What This
> Means For Jane "LLVM Developer" Doe' at the bottom.
>
> Motivation
> ----------
>
> We currently maintain two build systems (make, CMake). This has a couple of
> problems:
>
>   * Duplication: the two build systems duplicate a significant amount of implicit
>     logic (higher level dependencies), some config files, and other miscellaneous
>     features.
>
>   * Maintenance: Some parts of the build systems requires regular maintenance
>     (the CMake file lists, CMake dependency lists, module structure). Having one
>     more thing to maintain is annoying.
>
>   * Inconsistency: The two build systems behave inconsistently in some ways. If
>     we want both to officially supported systems, it would be nice for them to
>     behave as identically as possible.
>
>     For example, CMake now uses explicit dependencies which are hard coded into
>     the CMake files, but the Makefiles work completely differently.
>
> There are also other general issues with the way LLVM is built now:
>
>   * LLVM has a higher level structure for its organization, but this is not
>     explicit. LLVM is roughly organized into components (libraries, tools, etc.)
>     by directory. It would be nice to have this be more explicit.
>
>   * Much of the project build structure is implicit in the Makefiles or
>     CMakeFiles. It is not particularly explicit anywhere that, say, parts of
>     VMCore depend on building the Intrinsics, which depend on building tblgen.
>
>   * The Make system is not very efficient. We use recursive Makefiles which make
>     the current system (relatively) simple in some ways, but mean the Make build
>     is not nearly as scalable as it could be. In particular, the current
>     organization means the built is often serialized on something that is not a
>     strict dependency. It also makes it much more likely to do things like a
>     stampeding link of all the tools, even though many tools could have been
>     built earlier.
>
>
> Specific Goals
> --------------
>
>   * Move both build systems to use explicit library dependencies, in a clean
>     fashion. The CMake files do this now, but I don't think it has been made
>     clear to developers when they are supposed to edit these files, or how (other
>     than when something breaks, I guess).
>
>   * Explicitly describe as much of the project structure as necessary to support
>     builds. This means explicitly specifying how the project is organized, and
>     the dependencies among the components required to build (e.g., Intrinsics
>     before VMCore).
>
>     I believe a number of other projects/users (FreeBSD, fbuild) have
> built there own
>     build system for LLVM. Encoding the project structure clearly should make it
>     easier for such projects, or for other future users that want to do the same.
>
>     This should make it easier to experiment with the build system, for example
>     we could just directly generate good Makefiles for our project, or could
>     experiment with systems like Ninja which expect to be targetted by a
>     generator of some kind.
>
>
> Proposal
> --------
>
> My initial proposal is focused at moving us to use explicit library
> dependencies, but paves the way for centralizing more "build systemy" stuff in
> the future.
>
>   * Every LLVM "component" (roughly corresponds to each place we have a Makefile
>     or CMakeLists.txt currently) will get a 'LLVMBuild.txt' file.
>
>     This file will be an LLVM specific description of that component. Initially,
>     it will look something like this::
>
>       [component]
>       # The kind of component this is (currently library, tool, build tool). More
>       # types will be defined over time.
>       type = Library
>
>       # The name of the component.
>       name = VMCore
>
>       # The name of the component to logically group this in. This is just for
>       # organization purposes.
>       parent = Libraries
>
>       # The names of the library components that are also required when linking
>       # with this library. More on this later.
>       required_libraries = Support
>
>     The exact structure of the format is TBD (and easy to change), currently the
>     format is INI style but I may decide to change to JSON once all the pieces
>     are in place.
>
>     The LLVM web pages will have clear documentation on what these files should
>     look like, what is required, what is supported, and so on.
>
>
>   * I will add a new tool in utils/llvm-build which is designed to load and work
>     with these files. This tool will be written in Python, and the expectation is
>     that it can be run at configure time.
>
>     TO BE CLEAR: I intend to introduce a hard dependency on requiring Python in
>     order to build LLVM.
>
>     For the Makefiles, this is no worse than requiring Perl, so I don't think
>     there is much to argue with.
>
>     For CMake, this is a new dependency. However, I feel this is unavoidable:
>
>       * I see no way to support multiple build systems including CMake without
>         either introducing a dependency on some extra tool (which can be shared),
>         or duplicating a significant amount of logic in CMake.
>
>         I believe that duplicating logic is worse than adding the Python
>         dependency, and I think we already have more CMake code (a relatively
>         horrible language) than can be expected for most LLVM developers to deal
>         with.
>
>     Additionally, we already require Python for running our tests, so anyone
>     doing serious LLVM development should have it.
>
>     The one use case I believe this particularly hurts is users who just want to
>     download and play with LLVM, but I believe the Right Answer (tm) for that
>     case would be for us to provide nice installer packages anyway.
>
>
>   * utils/llvm-build will be run during configure time (both for Make and CMake),
>     and will:
>
>     * Generate the library dependency information required to link tools, in
>       whatever format makes the most system for the build system in use.
>
>     * Generate a C++ .inc file containing the dependency table for use by
>       llvm-config (which I am going to rewrite in C++).
>
>     * Add dependencies on the LLVMBuild.txt files to the build system, so that
>       the build will reconfigure appropriately when the library
> requirements change.
>
>
>   * Remove GenLibDeps.pl, find-cycles.pl, etc.
>
>     We will no longer be using these after llvm-config has moved over.
>
>
>   * Add explicit source file lists to the LLVMBuild.txt files. Unfortunately,
>     this is inevitable if we want to support CMake users projects automatically
>     reconfiguring themselves when new files get added. I can make it easier to
>     manage (for example, provide build targets that will automatically add any
>     new files).
>
>
>   * Move both Make and CMake over to using the explicit file lists in the
>     LLVMBuild files. This ensures that both systems get the same behavior
>     (instead of Make users being able to add files and forget to update
>     CMakeLists.txt).
>
>
>   * Add new 'lit' tests to check that the library dependency
> information is correct.
>
>     This seems a nicer place to do the checking which is currently partially
>     handled by find-cycles, and we should also be able to do a better job of the
>     checking (for example, verifying that the dependency list is "exact" -- only
>     specifies the minimum set of libraries required, and isn't allowed to specify
>     libraries which are only indirectly required).
>
>     It would be particularly cool if we could just write these tests using our
>     Object libraries.
>
>     This is one piece I haven't prototyped yet. I can obviously do something as
>     good as the current find-cycles.pl, but I hope to do better (eventually).
>
>
>   * These are just the first steps, after this I will continue to incrementally
>     try and move as much common information out of the Make and CMake systems so
>     there is "one source of truth" with regard to the project definition.
>
>
>   * I assume it is obvious, but when I say "LLVM" here I am referring to both
>     LLVM and Clang. I have not looked at lldb yet, but if it uses the LLVM build
>     system (and llvm-config) functionality I will try to make sure it
> continues to work.
>
>
> What This Means For Jane "LLVM Developer" Doe
> ---------------------------------------------
>
> In practice, this means:
>
>   * LLVM requires Python to build.
>
>   * When you add a file to LLVM, you will need to edit LLVMBuild.txt instead of
>     CMakeLists.txt, which will be in a slightly different, but otherwise totally
>     obvious format.
>
>     If you forget to do this, your file will not be built (which will most likely
>     cause a link error eventually). This is better than it being built by Make,
>     but causing CMake build failures when you check in.
>
>   * When you add a new library requirement to an existing component, you will be
>     required to edit LLVMBuild.txt instead of CMakeLists.txt, which will be in a
>     slightly different, but otherwise totally obvious (hopefully) format.
>
>     If you forget to do this, you will either get a link error or a test
>     failure. This is better than library you need transparently getting linked in
>     (with make) because it forces you to think about whether you actually should
>     be adding that dependency.
>
>     The goal is that this also ensures that if LLVM links and passes tests on
>     your system, then it should for everyone else as well.
>
>   * Developers not actively touching the build system should never need to touch
>     a Makefile or a CMake file.
>
> Overall, I believe this should be a quality of life improvement for the
> developer community. The only downside is having to deal with a new non-standard
> LLVM specific format, but I plan to solve this through documentation.
>
> Comments?
>
>   - Daniel
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

-- 
Marc J. Driftmeyer
Email :: mjd at reanimality.com <mailto:mjd at reanimality.com>
Web :: http://www.reanimality.com
Cell :: (509) 435-5212
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20111031/77553f29/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mjd.vcf
Type: text/x-vcard
Size: 317 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20111031/77553f29/attachment.vcf>