[LLVMdev] [RFC] Road map for CMake

Thu Jul 30 11:18:25 PDT 2015

> On Jul 30, 2015, at 11:11 AM, Reid Kleckner <rnk at google.com> wrote:
> 
> Cool, seems reasonable. That CMake feature request doesn't sound too onerous for them.
> 
> In the meantime, the Makefile generator will have more-or-less the right job scheduling due make's job server design. Most other generators already have bad scheduling, so it isn't as interesting to have one combined build.

Yes, the expectation is that Make’s job scheduling server makes changes like this unnecessary, and I suspect that by putting the Ninja sub-tasks into a limited size job pool we’ll get acceptable (but not optimal) scheduling.

-Chris

> 
> On Thu, Jul 30, 2015 at 10:37 AM, Chris Bieneman <beanz at apple.com <mailto:beanz at apple.com>> wrote:
> 
>> On Jul 30, 2015, at 10:30 AM, Reid Kleckner <rnk at google.com <mailto:rnk at google.com>> wrote:
>> 
>> On Wed, Jul 29, 2015 at 1:36 PM, Chris Bieneman <beanz at apple.com <mailto:beanz at apple.com>> wrote:
>> I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana).
>> 
>> Mono black devotion ftw! Thanks for working on this. :)
>> 
>> The main thing I'm trying to understand is, at each point along the way, will CMake be able to generate a combined build.ninja for all compilation jobs?
> 
> In order to generate a single combined build.ninja file we’ll actually need some changes to CMake. My intention is to have a high-level build file (generator agnostic) that will invoke the build tool for sub-projects for compiler-rt. For the Ninja generator we can use a custom job pool for the sub-projects to prevent over saturation of the build machine, but there will be sub-projects.
> 
> 
>> 
>> You're doing the work, so it's fine if that's a nongoal. However, I'd like it if the simple case where compiler-rt is being compiled for the host, the generated build files don't shell out to another build.
> 
> I agree, and that is how I would prefer this to work. I had a discussion with some of the guys from Sony a few weeks back about what it would take to be able to generate a single combined build.ninja file. There were three features we would need from CMake to make this work properly.
> 
> (1) Prefixing rules so that the sub-project rules wouldn’t conflict with each other or the master project’s rules
> (2) Importing the sub-projects into the parent ninja
> (3) Mark the CMake invocations for the external projects as “generator” rules so that Ninja would re-exec itself if CMake changed the build files
> 
> -Chris
> 
>> 
>> The main thing I want to talk about is PR21562. For Apple PR21562 is the biggest reason we can't fully abandon autoconf. I've spent some time over the last month hacking on compiler-rt's build system and I've come up with a basic outline of the approach I'd like to take.
>> 
>> (1) Reconcile out-of-tree functionality
>> 
>> Apple has a little bit of out-of-tree code that adds functionality to compiler-rt's CMake build system. Specifically the ability to build for iOS. Before I can do any substantial refactoring of the open source build system I need to bring it to feature parity with our internal system. I have three remaining patches for that.
>> 
>> D11083 [CMake] Add experimental support for building compiler-rt for iOS
>> D11082 [CMake] Adding some utility functions for Darwin builds into a new CompilerRTDarwinUtils.cmake module
>> D11073 Architectures for darwin need to be conditionalized based on the operating system.
>> 
>> One thing worth noting here. I need to land some variant of this functionality, but the way this is done is hacky and should go away. Once these are landed I can begin the more substantial reworking.
>> 
>> (2) Build the new Darwin build behavior
>> 
>> One of the big problems with building compiler-rt for Darwin is that the CMake has been hacked to build multiple platforms at once (yes I know my patches above only make this worse). The first thing I want to do is add a new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to toggle into a new mode for building on Darwin.
>> 
>> The new Darwin build mode will build a single triple at a time in much the same way the Android and Linux builds work. In fact, my plan is to have the new Darwin functionality almost completely follow the Linux and Android code paths.
>> 
>> There is a healthy bit of hand waving here because this task is larger than I'm making it sound. Compiler-rt is comprised of two very different sets of components, and they have some significantly different requirements. A big part of this work is going to be the root of PR21562, replacing the specific functionality in make/platform/clang_darwin.mk <http://clang_darwin.mk/>. That functionality drives building the runtime pieces of compiler-rt. There will need to be similar, but slightly different work done for the sanitizers.
>> 
>> (3) Make compiler-rt an external project when built in-tree with Clang
>> 
>> Once compiler-rt builds for one and only one target at a time, we will need to hook that into the LLVM build system so that you can do in-tree builds of compiler-rt for all the targets you can target with your new clang. I will gate this functionality on a flag just like the work above so it won't break existing users.
>> 
>> This re-working will also provide a more robust solution for PR14109. PR14109 can't be properly fixed without making compiler-rt an external project because CMake doesn't really support changing the compiler after you've already configured a directory.
>> 
>> Some of this functionality already exists in clang/runtime, but we'll want to transition to that being the only supported way to build compiler-rt, and (for Darwin) we'll need to add some support for lipo-ing thin binaries together.
>> 
>> (4) Celebrate!
>> 
>> That's really the hard bits. Once we get the new behavior working through the stack we'll need to test it like nobody's business and then we can talk about making it default and cleaning up all the code that supported the old way of doing things.
>> 
>> So... what's next?
>> 
>> With PR14109 and PR21562 done Apple will be able to migrate off autoconf, and there will only be a few currently identified issues preventing CMake from replacing autoconf. Tackling the remaining issues should be fairly straight forward as none of them are too gnarly.
>> 
>> Moving beyond this all I have some ideas for improving our CMake scripts to make developers more productive, and I have some ideas for improvements that we could get by working with the CMake community. For example, I want our CMake build system to have convenience targets for things that we currently drive with out-of-tree tooling. This includes clang bootstrap builds, and generating PGO data and linker order files. I also would like to work with the CMake community to get a way to wire up external projects so that they could all be mapped into a single Ninja build file for better parallelism and faster incremental builds.
>> 
>> Questions, comments, concerns, condolences?
>> 
>> Thanks,
>> -Chris
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu>         http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev <http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150730/6dd6a35b/attachment.html>