[llvm-dev] Building with LLVM_PARALLEL_XXX_JOBS
Sedat Dilek via llvm-dev
llvm-dev at lists.llvm.org
Wed Mar 2 23:09:18 PST 2016
I had only a quick view on the blog-texts.
It might be that a CLANG generated with LTO/PGO speeds up the build.
Can you confirm this?
Can you confirm binutils-gold speed up the build?
Has LLVM an own linker?
Can be used? Speedup the build?
Yesterday night I loooked through available CMAKE/LLVM variables...
### GOLD
# CMAKE_LINKER:FILEPATH=/usr/bin/ld
# GOLD_EXECUTABLE:FILEPATH=/usr/bin/ld.gold
# LLVM_TOOL_GOLD_BUILD:BOOL=ON
### OPTLEVEL
# CMAKE_ASM_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
# CMAKE_CXX_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
# CMAKE_C_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
### LTO
# LLVM_TOOL_LLVM_LTO_BUILD:BOOL=ON
# LLVM_TOOL_LTO_BUILD:BOOL=ON
### PGO
# LLVM_USE_OPROFILE:BOOL=OFF
#### TABLEGEN
# LLVM_OPTIMIZED_TABLEGEN:BOOL=OFF
So '-O3' is default for a RELEASE build.
Not sure which of the LTO variables are suitable, maybe both.
PGO? Is that the correct variable?
The blog-text mentioned to use optimized-tablegen.
Good? Bad? Ugly?
Thanks in advance for answering my questions.
Best regards,
- Sedat -
On 3/3/16, fariborz jahanian <fjahanian2016 at gmail.com> wrote:
> Building a binary with ‘LTO’, -O3, etc. will slow down the build. But the
> built binary could run much faster.
> I am not sure what the intention is here.
>
> - Fariborz
>
>> On Mar 2, 2016, at 4:42 PM, Mehdi Amini via llvm-dev
>> <llvm-dev at lists.llvm.org> wrote:
>>
>>
>>> On Mar 2, 2016, at 4:22 PM, Sedat Dilek <sedat.dilek at gmail.com> wrote:
>>>
>>> I got some more inspirations on how to speedup my build and integrated
>>> the URLs into my scripts (attached).
>>>
>>> For example to use GOLD as linker or to use '-O3' OptLevel maybe in
>>> combination with LTO and PGO (using '-O3 -flto -fprofile-use').
>>
>> LTO *will* slow down dramatically the build.
>>
>> --
>> Mehdi
>>
>>
>>
>>>
>>> Let's see when the v3.8.0 FINAL is released.
>>>
>>> - Sedat -
>>>
>>> On 3/2/16, Fabio Pagani <pagabuc at gmail.com> wrote:
>>>> Hey Chris,
>>>>
>>>> Sedat was asking for a way to "to speedup my build" and those blog
>>>> posts
>>>> were really helpful to me.
>>>> Anyway LLVM_DISTRIBUTION_COMPONENTS sounds very cool, hope you will
>>>> push
>>>> your code soon!
>>>>
>>>> On Tue, Mar 1, 2016 at 11:32 PM, Chris Bieneman <cbieneman at apple.com>
>>>> wrote:
>>>>
>>>>> Fabio, the work I was mentioning here is an extension beyond those
>>>>> blog
>>>>> posts.
>>>>>
>>>>> Some details:
>>>>>
>>>>> * The “almost 40%” number I referred to is a multi-stage clang build.
>>>>> That
>>>>> means we build a host-capable compiler, then build the actual compiler
>>>>> we
>>>>> want to ship.
>>>>> * I’m at Apple, so points 1 and 2 are already covered (we only use
>>>>> clang,
>>>>> and ld64 is a fast linker).
>>>>> * Our system compiler is PGO+LTO’d, but our stage1 isn’t. Stage1 isn’t
>>>>> because the performance improvement of PGO+LTO is less than the time
>>>>> it
>>>>> takes to build, and stage1 is basically a throwaway.
>>>>> * We are using Ninja and CMake, but this configuration isn’t really
>>>>> significantly faster than autoconf/make, and actually “ninja install”
>>>>> is
>>>>> slower in my tests than the old autoconf “make install”. The slowdown
>>>>> is
>>>>> almost entirely due to Ninja’s “all” target being a lot bigger.
>>>>> * This performance is for clean builds, not incremental so ccache or
>>>>> shared libraries would not be a valid optimization
>>>>> * We do use optimized tablegen
>>>>> * “Build Less” is exactly what the LLVM_DISTRIBUTION_COMPONENTS
>>>>> enables,
>>>>> just in a friendly wrapper target.
>>>>>
>>>>> -Chris
>>>>>
>>>>>
>>>>>
>>>>> On Mar 1, 2016, at 1:12 PM, Fabio Pagani <pagabuc at gmail.com> wrote:
>>>>>
>>>>> For faster builds and rebuilds you should definitely read:
>>>>>
>>>>> https://blogs.s-osg.org/an-introduction-to-accelerating-your-build-with-clang/
>>>>> https://blogs.s-osg.org/a-conclusion-to-accelerating-your-build-with-clang/
>>>>>
>>>>> Hope this helps!
>>>>>
>>>>> On Tue, Mar 1, 2016 at 9:17 PM, ChrisBieneman via llvm-dev <
>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>>> On Mar 1, 2016, at 10:01 AM, Mehdi Amini <mehdi.amini at apple.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>> On Mar 1, 2016, at 9:57 AM, Chris Bieneman <cbieneman at apple.com>
>>>>>> wrote:
>>>>>>>>
>>>>>>>> There are a few notes I'd like to add to this thread.
>>>>>>>>
>>>>>>>> (1) we have a number of places throughout out CMake build where we
>>>>>>>> use
>>>>>> features from newer CMakes gated by version checks. Most of these
>>>>>> features
>>>>>> are performance or usability related. None of them are correctness.
>>>>>> Using
>>>>>> the latest CMake release will often result in faster builds, so I
>>>>>> encourage
>>>>>> it.
>>>>>>>>
>>>>>>>> (2) CMake's "install" target will pretty much always be slower from
>>>>>> clean than the old autoconf/make "install" target. This is because in
>>>>>> CMake
>>>>>> "install" depends on "all", and our CMake builds more stuff in "all"
>>>>>> than
>>>>>> autoconf did. To help with this or CMake system has lots of
>>>>>> convenient
>>>>>> "install-${name}" targets that support component-based installation.
>>>>>> Not
>>>>>> every component has one of these rules, but if one you need is
>>>>>> missing
>>>>>> let
>>>>>> me know. I also recently (r261681) added a new option
>>>>>> (LLVM_DISTRIBUTION_COMPONENTS) that allows you to specify a list of
>>>>>> components that have custom install targets. It then creates a new
>>>>>> "install-distribution" target that wraps just the components you
>>>>>> want.
>>>>>> For
>>>>>> Apple this is almost a 40% speed up over "ninja install".
>>>>>>>
>>>>>>> That sounds great, I want to use it!
>>>>>>> It would even be more awesome with an description/example in
>>>>>> docs/CMake.rst :)
>>>>>>
>>>>>> Once I get the last of the kinks worked out for our internal adoption
>>>>>> I'm
>>>>>> going to open source our config files that use it.
>>>>>>
>>>>>> I've also made a note to remind myself to document it in
>>>>>> docs/CMake.rst.
>>>>>> I need to do a pass updating that with a bunch of the cool new things
>>>>>> we're
>>>>>> doing with CMake. Thanks for the reminder.
>>>>>>
>>>>>> -Chris
>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Mehdi
>>>>>>>>
>>>>>>>> On Feb 25, 2016, at 11:08 AM, Sedat Dilek via llvm-dev <
>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>>>
>>>>>>>>>> Which combination of cmake/ninja versions are you using (latest
>>>>>>>>>> are
>>>>>>>>>> v3.4.3 and v1.6.0)?
>>>>>>>>>
>>>>>>>>> With this combination I could reduce build-time down from approx.
>>>>>>>>> 3h
>>>>>>>>> down to 01h20m.
>>>>>>>>>
>>>>>>>>> $ egrep -i 'jobs|ninja' llvm-build/CMakeCache.txt
>>>>>>>>> //Program used to build from build.ninja files.
>>>>>>>>> CMAKE_MAKE_PROGRAM:FILEPATH=/opt/cmake/bin/ninja
>>>>>>>>> //Define the maximum number of concurrent compilation jobs.
>>>>>>>>> LLVM_PARALLEL_COMPILE_JOBS:STRING=3
>>>>>>>>> //Define the maximum number of concurrent link jobs.
>>>>>>>>> LLVM_PARALLEL_LINK_JOBS:STRING=1
>>>>>>>>> CMAKE_GENERATOR:INTERNAL=Ninja
>>>>>>>>>
>>>>>>>>> $ LC_ALL=C ls -alt
>>>>>> logs/3.8.0rc3_clang-3-8-0-rc3_cmake-3-4-3_ninja-1-6-0/
>>>>>>>>> total 360
>>>>>>>>> drwxr-xr-x 2 wearefam wearefam 4096 Feb 25 19:58 .
>>>>>>>>> drwxr-xr-x 6 wearefam wearefam 4096 Feb 25 19:58 ..
>>>>>>>>> -rw-r--r-- 1 wearefam wearefam 130196 Feb 25 19:54
>>>>>>>>> install-log_llvm-toolchain-3.8.0rc3.txt
>>>>>>>>> -rw-r--r-- 1 wearefam wearefam 205762 Feb 25 19:51
>>>>>>>>> build-log_llvm-toolchain-3.8.0rc3.txt
>>>>>>>>> -rw-r--r-- 1 wearefam wearefam 14331 Feb 25 18:30
>>>>>>>>> configure-log_llvm-toolchain-3.8.0rc3.txt
>>>>>>>>>
>>>>>>>>> $ LC_ALL=C du -s -m llvm* /opt/llvm-toolchain-3.8.0rc3
>>>>>>>>> 315 llvm
>>>>>>>>> 941 llvm-build
>>>>>>>>> 609 /opt/llvm-toolchain-3.8.0rc3
>>>>>>>>>
>>>>>>>>> - Sedat -
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://cmake.org/files/v3.5/cmake-3.5.0-rc3-Linux-x86_64.tar.gz
>>>>>>>>> _______________________________________________
>>>>>>>>> LLVM Developers mailing list
>>>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>
>>>>>> _______________________________________________
>>>>>> LLVM Developers mailing list
>>>>>> llvm-dev at lists.llvm.org
>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>> <build_llvm-toolchain_clang-cmake-ninja.sh><install_llvm-toolchain_clang-cmake-ninja.sh>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
More information about the llvm-dev
mailing list