[llvm-dev] Building with LLVM_PARALLEL_XXX_JOBS

fariborz jahanian via llvm-dev llvm-dev at lists.llvm.org
Wed Mar 2 16:52:54 PST 2016 

Building a binary with ‘LTO’, -O3, etc.  will slow down the build. But the built binary could run much faster.
I am not sure what the intention is here.

- Fariborz

> On Mar 2, 2016, at 4:42 PM, Mehdi Amini via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> 
>> On Mar 2, 2016, at 4:22 PM, Sedat Dilek <sedat.dilek at gmail.com> wrote:
>> 
>> I got some more inspirations on how to speedup my build and integrated
>> the URLs into my scripts (attached).
>> 
>> For example to use GOLD as linker or to use '-O3' OptLevel maybe in
>> combination with LTO and PGO (using '-O3 -flto -fprofile-use').
> 
> LTO *will* slow down dramatically the build.
> 
> -- 
> Mehdi
> 
> 
> 
>> 
>> Let's see when the v3.8.0 FINAL is released.
>> 
>> - Sedat -
>> 
>> On 3/2/16, Fabio Pagani <pagabuc at gmail.com> wrote:
>>> Hey Chris,
>>> 
>>> Sedat was asking for a way to "to speedup my build" and those blog posts
>>> were really helpful to me.
>>> Anyway LLVM_DISTRIBUTION_COMPONENTS sounds very cool, hope you will push
>>> your code soon!
>>> 
>>> On Tue, Mar 1, 2016 at 11:32 PM, Chris Bieneman <cbieneman at apple.com>
>>> wrote:
>>> 
>>>> Fabio, the work I was mentioning here is an extension beyond those blog
>>>> posts.
>>>> 
>>>> Some details:
>>>> 
>>>> * The “almost 40%” number I referred to is a multi-stage clang build.
>>>> That
>>>> means we build a host-capable compiler, then build the actual compiler we
>>>> want to ship.
>>>> * I’m at Apple, so points 1 and 2 are already covered (we only use clang,
>>>> and ld64 is a fast linker).
>>>> * Our system compiler is PGO+LTO’d, but our stage1 isn’t. Stage1 isn’t
>>>> because the performance improvement of PGO+LTO is less than the time it
>>>> takes to build, and stage1 is basically a throwaway.
>>>> * We are using Ninja and CMake, but this configuration isn’t really
>>>> significantly faster than autoconf/make, and actually “ninja install” is
>>>> slower in my tests than the old autoconf “make install”. The slowdown is
>>>> almost entirely due to Ninja’s “all” target being a lot bigger.
>>>> * This performance is for clean builds, not incremental so ccache or
>>>> shared libraries would not be a valid optimization
>>>> * We do use optimized tablegen
>>>> * “Build Less” is exactly what the LLVM_DISTRIBUTION_COMPONENTS enables,
>>>> just in a friendly wrapper target.
>>>> 
>>>> -Chris
>>>> 
>>>> 
>>>> 
>>>> On Mar 1, 2016, at 1:12 PM, Fabio Pagani <pagabuc at gmail.com> wrote:
>>>> 
>>>> For faster builds and rebuilds you should definitely read:
>>>> 
>>>> https://blogs.s-osg.org/an-introduction-to-accelerating-your-build-with-clang/
>>>> https://blogs.s-osg.org/a-conclusion-to-accelerating-your-build-with-clang/
>>>> 
>>>> Hope this helps!
>>>> 
>>>> On Tue, Mar 1, 2016 at 9:17 PM, ChrisBieneman via llvm-dev <
>>>> llvm-dev at lists.llvm.org> wrote:
>>>> 
>>>>> 
>>>>> 
>>>>>> On Mar 1, 2016, at 10:01 AM, Mehdi Amini <mehdi.amini at apple.com>
>>>>>> wrote:
>>>>>> 
>>>>>> 
>>>>>>> On Mar 1, 2016, at 9:57 AM, Chris Bieneman <cbieneman at apple.com>
>>>>> wrote:
>>>>>>> 
>>>>>>> There are a few notes I'd like to add to this thread.
>>>>>>> 
>>>>>>> (1) we have a number of places throughout out CMake build where we
>>>>>>> use
>>>>> features from newer CMakes gated by version checks. Most of these
>>>>> features
>>>>> are performance or usability related. None of them are correctness.
>>>>> Using
>>>>> the latest CMake release will often result in faster builds, so I
>>>>> encourage
>>>>> it.
>>>>>>> 
>>>>>>> (2) CMake's "install" target will pretty much always be slower from
>>>>> clean than the old autoconf/make "install" target. This is because in
>>>>> CMake
>>>>> "install" depends on "all", and our CMake builds more stuff in "all"
>>>>> than
>>>>> autoconf did. To help with this or CMake system has lots of convenient
>>>>> "install-${name}" targets that support component-based installation. Not
>>>>> every component has one of these rules, but if one you need is missing
>>>>> let
>>>>> me know. I also recently (r261681) added a new option
>>>>> (LLVM_DISTRIBUTION_COMPONENTS) that allows you to specify a list of
>>>>> components that have custom install targets. It then creates a new
>>>>> "install-distribution" target that wraps just the components you want.
>>>>> For
>>>>> Apple this is almost a 40% speed up over "ninja install".
>>>>>> 
>>>>>> That sounds great, I want to use it!
>>>>>> It would even be more awesome with an description/example in
>>>>> docs/CMake.rst :)
>>>>> 
>>>>> Once I get the last of the kinks worked out for our internal adoption
>>>>> I'm
>>>>> going to open source our config files that use it.
>>>>> 
>>>>> I've also made a note to remind myself to document it in docs/CMake.rst.
>>>>> I need to do a pass updating that with a bunch of the cool new things
>>>>> we're
>>>>> doing with CMake. Thanks for the reminder.
>>>>> 
>>>>> -Chris
>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Mehdi
>>>>>>> 
>>>>>>> On Feb 25, 2016, at 11:08 AM, Sedat Dilek via llvm-dev <
>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>> 
>>>>>>>>> Which combination of cmake/ninja versions are you using (latest are
>>>>>>>>> v3.4.3 and v1.6.0)?
>>>>>>>> 
>>>>>>>> With this combination I could reduce build-time down from approx. 3h
>>>>>>>> down to 01h20m.
>>>>>>>> 
>>>>>>>> $ egrep -i 'jobs|ninja' llvm-build/CMakeCache.txt
>>>>>>>> //Program used to build from build.ninja files.
>>>>>>>> CMAKE_MAKE_PROGRAM:FILEPATH=/opt/cmake/bin/ninja
>>>>>>>> //Define the maximum number of concurrent compilation jobs.
>>>>>>>> LLVM_PARALLEL_COMPILE_JOBS:STRING=3
>>>>>>>> //Define the maximum number of concurrent link jobs.
>>>>>>>> LLVM_PARALLEL_LINK_JOBS:STRING=1
>>>>>>>> CMAKE_GENERATOR:INTERNAL=Ninja
>>>>>>>> 
>>>>>>>> $ LC_ALL=C ls -alt
>>>>> logs/3.8.0rc3_clang-3-8-0-rc3_cmake-3-4-3_ninja-1-6-0/
>>>>>>>> total 360
>>>>>>>> drwxr-xr-x 2 wearefam wearefam   4096 Feb 25 19:58 .
>>>>>>>> drwxr-xr-x 6 wearefam wearefam   4096 Feb 25 19:58 ..
>>>>>>>> -rw-r--r-- 1 wearefam wearefam 130196 Feb 25 19:54
>>>>>>>> install-log_llvm-toolchain-3.8.0rc3.txt
>>>>>>>> -rw-r--r-- 1 wearefam wearefam 205762 Feb 25 19:51
>>>>>>>> build-log_llvm-toolchain-3.8.0rc3.txt
>>>>>>>> -rw-r--r-- 1 wearefam wearefam  14331 Feb 25 18:30
>>>>>>>> configure-log_llvm-toolchain-3.8.0rc3.txt
>>>>>>>> 
>>>>>>>> $ LC_ALL=C du -s -m llvm* /opt/llvm-toolchain-3.8.0rc3
>>>>>>>> 315     llvm
>>>>>>>> 941     llvm-build
>>>>>>>> 609     /opt/llvm-toolchain-3.8.0rc3
>>>>>>>> 
>>>>>>>> - Sedat -
>>>>>>>> 
>>>>>>>> [1] https://cmake.org/files/v3.5/cmake-3.5.0-rc3-Linux-x86_64.tar.gz
>>>>>>>> _______________________________________________
>>>>>>>> LLVM Developers mailing list
>>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>> 
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>> <build_llvm-toolchain_clang-cmake-ninja.sh><install_llvm-toolchain_clang-cmake-ninja.sh>
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

More information about the llvm-dev mailing list