[llvm-dev] Building with LLVM_PARALLEL_XXX_JOBS

Wed Mar 2 13:28:02 PST 2016

Hey Chris,

Sedat was asking for a way to "to speedup my build" and those blog posts
were really helpful to me.
Anyway LLVM_DISTRIBUTION_COMPONENTS sounds very cool, hope you will push
your code soon!

On Tue, Mar 1, 2016 at 11:32 PM, Chris Bieneman <cbieneman at apple.com> wrote:

> Fabio, the work I was mentioning here is an extension beyond those blog
> posts.
>
> Some details:
>
> * The “almost 40%” number I referred to is a multi-stage clang build. That
> means we build a host-capable compiler, then build the actual compiler we
> want to ship.
> * I’m at Apple, so points 1 and 2 are already covered (we only use clang,
> and ld64 is a fast linker).
> * Our system compiler is PGO+LTO’d, but our stage1 isn’t. Stage1 isn’t
> because the performance improvement of PGO+LTO is less than the time it
> takes to build, and stage1 is basically a throwaway.
> * We are using Ninja and CMake, but this configuration isn’t really
> significantly faster than autoconf/make, and actually “ninja install” is
> slower in my tests than the old autoconf “make install”. The slowdown is
> almost entirely due to Ninja’s “all” target being a lot bigger.
> * This performance is for clean builds, not incremental so ccache or
> shared libraries would not be a valid optimization
> * We do use optimized tablegen
> * “Build Less” is exactly what the LLVM_DISTRIBUTION_COMPONENTS enables,
> just in a friendly wrapper target.
>
> -Chris
>
>
>
> On Mar 1, 2016, at 1:12 PM, Fabio Pagani <pagabuc at gmail.com> wrote:
>
> For faster builds and rebuilds you should definitely read:
>
> https://blogs.s-osg.org/an-introduction-to-accelerating-your-build-with-clang/
> https://blogs.s-osg.org/a-conclusion-to-accelerating-your-build-with-clang/
>
> Hope this helps!
>
> On Tue, Mar 1, 2016 at 9:17 PM, ChrisBieneman via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>>
>>
>> > On Mar 1, 2016, at 10:01 AM, Mehdi Amini <mehdi.amini at apple.com> wrote:
>> >
>> >
>> >> On Mar 1, 2016, at 9:57 AM, Chris Bieneman <cbieneman at apple.com>
>> wrote:
>> >>
>> >> There are a few notes I'd like to add to this thread.
>> >>
>> >> (1) we have a number of places throughout out CMake build where we use
>> features from newer CMakes gated by version checks. Most of these features
>> are performance or usability related. None of them are correctness. Using
>> the latest CMake release will often result in faster builds, so I encourage
>> it.
>> >>
>> >> (2) CMake's "install" target will pretty much always be slower from
>> clean than the old autoconf/make "install" target. This is because in CMake
>> "install" depends on "all", and our CMake builds more stuff in "all" than
>> autoconf did. To help with this or CMake system has lots of convenient
>> "install-${name}" targets that support component-based installation. Not
>> every component has one of these rules, but if one you need is missing let
>> me know. I also recently (r261681) added a new option
>> (LLVM_DISTRIBUTION_COMPONENTS) that allows you to specify a list of
>> components that have custom install targets. It then creates a new
>> "install-distribution" target that wraps just the components you want. For
>> Apple this is almost a 40% speed up over "ninja install".
>> >
>> > That sounds great, I want to use it!
>> > It would even be more awesome with an description/example in
>> docs/CMake.rst :)
>>
>> Once I get the last of the kinks worked out for our internal adoption I'm
>> going to open source our config files that use it.
>>
>> I've also made a note to remind myself to document it in docs/CMake.rst.
>> I need to do a pass updating that with a bunch of the cool new things we're
>> doing with CMake. Thanks for the reminder.
>>
>> -Chris
>>
>> >
>> > --
>> > Mehdi
>> >>
>> >> On Feb 25, 2016, at 11:08 AM, Sedat Dilek via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>> >>
>> >>>> Which combination of cmake/ninja versions are you using (latest are
>> >>>> v3.4.3 and v1.6.0)?
>> >>>
>> >>> With this combination I could reduce build-time down from approx. 3h
>> >>> down to 01h20m.
>> >>>
>> >>> $ egrep -i 'jobs|ninja' llvm-build/CMakeCache.txt
>> >>> //Program used to build from build.ninja files.
>> >>> CMAKE_MAKE_PROGRAM:FILEPATH=/opt/cmake/bin/ninja
>> >>> //Define the maximum number of concurrent compilation jobs.
>> >>> LLVM_PARALLEL_COMPILE_JOBS:STRING=3
>> >>> //Define the maximum number of concurrent link jobs.
>> >>> LLVM_PARALLEL_LINK_JOBS:STRING=1
>> >>> CMAKE_GENERATOR:INTERNAL=Ninja
>> >>>
>> >>> $ LC_ALL=C ls -alt
>> logs/3.8.0rc3_clang-3-8-0-rc3_cmake-3-4-3_ninja-1-6-0/
>> >>> total 360
>> >>> drwxr-xr-x 2 wearefam wearefam   4096 Feb 25 19:58 .
>> >>> drwxr-xr-x 6 wearefam wearefam   4096 Feb 25 19:58 ..
>> >>> -rw-r--r-- 1 wearefam wearefam 130196 Feb 25 19:54
>> >>> install-log_llvm-toolchain-3.8.0rc3.txt
>> >>> -rw-r--r-- 1 wearefam wearefam 205762 Feb 25 19:51
>> >>> build-log_llvm-toolchain-3.8.0rc3.txt
>> >>> -rw-r--r-- 1 wearefam wearefam  14331 Feb 25 18:30
>> >>> configure-log_llvm-toolchain-3.8.0rc3.txt
>> >>>
>> >>> $ LC_ALL=C du -s -m llvm* /opt/llvm-toolchain-3.8.0rc3
>> >>> 315     llvm
>> >>> 941     llvm-build
>> >>> 609     /opt/llvm-toolchain-3.8.0rc3
>> >>>
>> >>> - Sedat -
>> >>>
>> >>> [1] https://cmake.org/files/v3.5/cmake-3.5.0-rc3-Linux-x86_64.tar.gz
>> >>> _______________________________________________
>> >>> LLVM Developers mailing list
>> >>> llvm-dev at lists.llvm.org
>> >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> >
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160302/67aa4df3/attachment.html>