[cfe-dev] RFC: Upcoming Build System Changes

[Nico Weber]

Hi,

my 2 cents:

1) If you're targeting python 2.6 or less, I prefer a python
dependency over cmake 'cause it's already installed on all machines I
build clang on. If you're targeting 2.7+, I don't care either way.

2) On the chromium project, we're using a custom meta build system
written in python, and it's been a lot more useful than we initially
expected. Based on this experience, your proposal sounds good to me.

Nico

On Thu, Oct 27, 2011 at 6:11 PM, Daniel Dunbar <daniel at zuster.org> wrote:
> Hi all,
>
> As you might have inferred, I'm in the process of working on some changes to the
> way LLVM builds. I have outlined a rough proposal below, unless there are any
> major objections I will probably start committing stuff next week.
>
> This message may be verbose, if you want the executive summary, skip
> to 'What This
> Means For Jane "LLVM Developer" Doe' at the bottom.
>
> Motivation
> ----------
>
> We currently maintain two build systems (make, CMake). This has a couple of
> problems:
>
>  * Duplication: the two build systems duplicate a significant amount of implicit
>   logic (higher level dependencies), some config files, and other miscellaneous
>   features.
>
>  * Maintenance: Some parts of the build systems requires regular maintenance
>   (the CMake file lists, CMake dependency lists, module structure). Having one
>   more thing to maintain is annoying.
>
>  * Inconsistency: The two build systems behave inconsistently in some ways. If
>   we want both to officially supported systems, it would be nice for them to
>   behave as identically as possible.
>
>   For example, CMake now uses explicit dependencies which are hard coded into
>   the CMake files, but the Makefiles work completely differently.
>
> There are also other general issues with the way LLVM is built now:
>
>  * LLVM has a higher level structure for its organization, but this is not
>   explicit. LLVM is roughly organized into components (libraries, tools, etc.)
>   by directory. It would be nice to have this be more explicit.
>
>  * Much of the project build structure is implicit in the Makefiles or
>   CMakeFiles. It is not particularly explicit anywhere that, say, parts of
>   VMCore depend on building the Intrinsics, which depend on building tblgen.
>
>  * The Make system is not very efficient. We use recursive Makefiles which make
>   the current system (relatively) simple in some ways, but mean the Make build
>   is not nearly as scalable as it could be. In particular, the current
>   organization means the built is often serialized on something that is not a
>   strict dependency. It also makes it much more likely to do things like a
>   stampeding link of all the tools, even though many tools could have been
>   built earlier.
>
>
> Specific Goals
> --------------
>
>  * Move both build systems to use explicit library dependencies, in a clean
>   fashion. The CMake files do this now, but I don't think it has been made
>   clear to developers when they are supposed to edit these files, or how (other
>   than when something breaks, I guess).
>
>  * Explicitly describe as much of the project structure as necessary to support
>   builds. This means explicitly specifying how the project is organized, and
>   the dependencies among the components required to build (e.g., Intrinsics
>   before VMCore).
>
>   I believe a number of other projects/users (FreeBSD, fbuild) have
> built there own
>   build system for LLVM. Encoding the project structure clearly should make it
>   easier for such projects, or for other future users that want to do the same.
>
>   This should make it easier to experiment with the build system, for example
>   we could just directly generate good Makefiles for our project, or could
>   experiment with systems like Ninja which expect to be targetted by a
>   generator of some kind.
>
>
> Proposal
> --------
>
> My initial proposal is focused at moving us to use explicit library
> dependencies, but paves the way for centralizing more "build systemy" stuff in
> the future.
>
>  * Every LLVM "component" (roughly corresponds to each place we have a Makefile
>   or CMakeLists.txt currently) will get a 'LLVMBuild.txt' file.
>
>   This file will be an LLVM specific description of that component. Initially,
>   it will look something like this::
>
>     [component]
>     # The kind of component this is (currently library, tool, build tool). More
>     # types will be defined over time.
>     type = Library
>
>     # The name of the component.
>     name = VMCore
>
>     # The name of the component to logically group this in. This is just for
>     # organization purposes.
>     parent = Libraries
>
>     # The names of the library components that are also required when linking
>     # with this library. More on this later.
>     required_libraries = Support
>
>   The exact structure of the format is TBD (and easy to change), currently the
>   format is INI style but I may decide to change to JSON once all the pieces
>   are in place.
>
>   The LLVM web pages will have clear documentation on what these files should
>   look like, what is required, what is supported, and so on.
>
>
>  * I will add a new tool in utils/llvm-build which is designed to load and work
>   with these files. This tool will be written in Python, and the expectation is
>   that it can be run at configure time.
>
>   TO BE CLEAR: I intend to introduce a hard dependency on requiring Python in
>   order to build LLVM.
>
>   For the Makefiles, this is no worse than requiring Perl, so I don't think
>   there is much to argue with.
>
>   For CMake, this is a new dependency. However, I feel this is unavoidable:
>
>     * I see no way to support multiple build systems including CMake without
>       either introducing a dependency on some extra tool (which can be shared),
>       or duplicating a significant amount of logic in CMake.
>
>       I believe that duplicating logic is worse than adding the Python
>       dependency, and I think we already have more CMake code (a relatively
>       horrible language) than can be expected for most LLVM developers to deal
>       with.
>
>   Additionally, we already require Python for running our tests, so anyone
>   doing serious LLVM development should have it.
>
>   The one use case I believe this particularly hurts is users who just want to
>   download and play with LLVM, but I believe the Right Answer (tm) for that
>   case would be for us to provide nice installer packages anyway.
>
>
>  * utils/llvm-build will be run during configure time (both for Make and CMake),
>   and will:
>
>   * Generate the library dependency information required to link tools, in
>     whatever format makes the most system for the build system in use.
>
>   * Generate a C++ .inc file containing the dependency table for use by
>     llvm-config (which I am going to rewrite in C++).
>
>   * Add dependencies on the LLVMBuild.txt files to the build system, so that
>     the build will reconfigure appropriately when the library
> requirements change.
>
>
>  * Remove GenLibDeps.pl, find-cycles.pl, etc.
>
>   We will no longer be using these after llvm-config has moved over.
>
>
>  * Add explicit source file lists to the LLVMBuild.txt files. Unfortunately,
>   this is inevitable if we want to support CMake users projects automatically
>   reconfiguring themselves when new files get added. I can make it easier to
>   manage (for example, provide build targets that will automatically add any
>   new files).
>
>
>  * Move both Make and CMake over to using the explicit file lists in the
>   LLVMBuild files. This ensures that both systems get the same behavior
>   (instead of Make users being able to add files and forget to update
>   CMakeLists.txt).
>
>
>  * Add new 'lit' tests to check that the library dependency
> information is correct.
>
>   This seems a nicer place to do the checking which is currently partially
>   handled by find-cycles, and we should also be able to do a better job of the
>   checking (for example, verifying that the dependency list is "exact" -- only
>   specifies the minimum set of libraries required, and isn't allowed to specify
>   libraries which are only indirectly required).
>
>   It would be particularly cool if we could just write these tests using our
>   Object libraries.
>
>   This is one piece I haven't prototyped yet. I can obviously do something as
>   good as the current find-cycles.pl, but I hope to do better (eventually).
>
>
>  * These are just the first steps, after this I will continue to incrementally
>   try and move as much common information out of the Make and CMake systems so
>   there is "one source of truth" with regard to the project definition.
>
>
>  * I assume it is obvious, but when I say "LLVM" here I am referring to both
>   LLVM and Clang. I have not looked at lldb yet, but if it uses the LLVM build
>   system (and llvm-config) functionality I will try to make sure it
> continues to work.
>
>
> What This Means For Jane "LLVM Developer" Doe
> ---------------------------------------------
>
> In practice, this means:
>
>  * LLVM requires Python to build.
>
>  * When you add a file to LLVM, you will need to edit LLVMBuild.txt instead of
>   CMakeLists.txt, which will be in a slightly different, but otherwise totally
>   obvious format.
>
>   If you forget to do this, your file will not be built (which will most likely
>   cause a link error eventually). This is better than it being built by Make,
>   but causing CMake build failures when you check in.
>
>  * When you add a new library requirement to an existing component, you will be
>   required to edit LLVMBuild.txt instead of CMakeLists.txt, which will be in a
>   slightly different, but otherwise totally obvious (hopefully) format.
>
>   If you forget to do this, you will either get a link error or a test
>   failure. This is better than library you need transparently getting linked in
>   (with make) because it forces you to think about whether you actually should
>   be adding that dependency.
>
>   The goal is that this also ensures that if LLVM links and passes tests on
>   your system, then it should for everyone else as well.
>
>  * Developers not actively touching the build system should never need to touch
>   a Makefile or a CMake file.
>
> Overall, I believe this should be a quality of life improvement for the
> developer community. The only downside is having to deal with a new non-standard
> LLVM specific format, but I plan to solve this through documentation.
>
> Comments?
>
>  - Daniel
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>