[llvm-dev] Building with LLVM_PARALLEL_XXX_JOBS

Sat Mar 12 04:04:42 PST 2016

On Sat, Mar 12, 2016 at 12:45 PM, Sedat Dilek <sedat.dilek at gmail.com> wrote:
> On Fri, Mar 4, 2016 at 11:28 AM, Tilmann Scheller
> <tilmann at osg.samsung.com> wrote:
>> Hi Sedat,
>>
>> On 03/03/2016 08:09 AM, Sedat Dilek via llvm-dev wrote:
>>>
>>> It might be that a CLANG generated with LTO/PGO speeds up the build.
>>> Can you confirm this?
>>
>> Yes, a Clang host compiler built with LTO or PGO is generally faster than an
>> -O3 build.
>>
>> Some things to keep in mind when building the Clang host compiler:
>>
>> GCC:
>>   - GCC 4.9 gives good results with PGO enabled (1.16x speedup over the -O3
>> build), not so much with LTO (actually regresses performance over the -O3
>> build, same for PGO vs PGO+LTO)
>>   - GCC 5.1/5.2/5.3 can't build Clang with LTO enabled
>> (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66027), that's supposed to be
>> fixed in GCC 5.4
>>
>> Clang:
>>   - PGO works and gives a good 1.12x speedup over the -O3 build (produced
>> about 270GB of profiling data when I tried this in December last year, this
>> should be addressed soon once the in-process profiling data merging lands)
>>   - LTO provides a 1.03x speedup over the -O3 build
>>   - I have not tried LTO+PGO with full Clang bootstrap profiling data but I
>> would expect that it helps to increase the performance even further
>>
>>> Can you confirm binutils-gold speed up the build?
>>
>> Yes, gold is definitely faster than ld when building Clang/LLVM.
>>
>>> Has LLVM an own linker?
>>> Can be used? Speedup the build?
>>
>> I haven't tried it but lld can definitely link Clang/LLVM on x86-64 Linux.
>>
>>> The blog-text mentioned to use optimized-tablegen.
>>> Good? Bad? Ugly?
>>
>> Good, it helps to speed up debug builds.
>>
>
> [ CCed all folks who answered me ]
>
> Hi,
>
> I have built my llvm-toolchain v3.8.0 (FINAL) with binutils-gold v1.11
> in a 1st run.
>
> When building with cmake/ninja there are 150 "Linking" lines...
>
> $ grep Linking logs/3.8.0_clang-3-8-0_cmake-3-4-3_ninja-1-6-0_gold-1-11_compile-jobs-2_link-jobs-1/build-log_llvm-toolchain-3.8.0.txt
> | wc -l
> 150
>
> I have the following cmake-options realized...
>
> *** SNIP ***
>
> ### CMAKE OPTIONS
> # NOTE #1: CMake Version 2.8.8 is the minimum required(Ubuntu/precise
> ships v2.8.7 officially)
> # NOTE #2: For fast builds use recommended CMake >= v3.2 (used:
> v3.4.3) and Ninja (used: v1.6.0)
> # NOTE #3: How to check available cmake-options?
> # EXAMPLE #3: cd $BUILD_DIR ; cmake ../llvm -LA | egrep $CMAKE_OPTS
> #
> # CMake binary
> CMAKE="cmake"
> # CMake compiler options
> COMPILERS_CMAKE_OPTS="-DCMAKE_C_COMPILER=$COMPILER_CC
> -DCMAKE_CXX_COMPILER=$COMPILER_CXX"
> # NOTE-1: cmake/ninja: Use LLVM_PARALLEL_COMPILE_JOBS and
> LLVM_PARALLEL_LINK_JOBS options
> # NOTE-2: For fast builds use available (online) CPUs +1 or set values
> explicitly
> # NOTE-3: For fast and safe linking use bintils-gold and LINK_JOBS="1"
> COMPILE_JOBS="2"
> ##COMPILE_JOBS=$(($(getconf _NPROCESSORS_ONLN)+1))
> LINK_JOBS="1"
> JOBS_CMAKE_OPTS="-DLLVM_PARALLEL_COMPILE_JOBS=$COMPILE_JOBS
> -DLLVM_PARALLEL_LINK_JOBS=$LINK_JOBS"
> # Cmake linker options (here: Use binutils-gold to speedup build)
> LINKER="/usr/bin/ld.gold"
> CMAKE_LINKER="$LINKER"
> CMAKE_LINKER_OPTS="-DCMAKE_LINKER=$CMAKE_LINKER"
> # CMake Generators
> CMAKE_GENERATORS="Ninja"
> GENERATORS_CMAKE_OPTS="-G $CMAKE_GENERATORS"
> # CMake configure settings
> PREFIX_CMAKE_OPTS="-DCMAKE_INSTALL_PREFIX=$PREFIX"
> OPTIMIZED_CMAKE_OPTS="-DCMAKE_BUILD_TYPE=RELEASE"
> ASSERTIONS_CMAKE_OPTS="-DLLVM_ENABLE_ASSERTIONS=ON"
> TARGETS_CMAKE_OPTS="-DLLVM_TARGETS_TO_BUILD=X86"
> CONFIGURE_CMAKE_OPTS="$PREFIX_CMAKE_OPTS $OPTIMIZED_CMAKE_OPTS
> $ASSERTIONS_CMAKE_OPTS $TARGETS_CMAKE_OPTS"
> # All CMake options
> CMAKE_OPTS="$COMPILERS_CMAKE_OPTS $JOBS_CMAKE_OPTS $CMAKE_LINKER_OPTS
> $GENERATORS_CMAKE_OPTS $CONFIGURE_CMAKE_OPTS"
>
> *** SNAP ***
>
> Is LINK_JOBS="2" speeding up the build?
> One guy of you told me to use "1" to be on the safe side - that is my default.
>
> Personally, I do not think this is very very efficiently - more than
> with binutild-bfd Linker.
>
> Using llvm-link (in a 2nd build) - good|bad|ugly?
>
> [ TODO#S: Before doing a 2nd build (and in a 3rd run using more
> optimized binaries) ]
>
> How do I anable LTO via CMAKE?
> How do I enable PGO via CMAKE?
>
> Grepping for 'lto' 'pgo' gives no help in [1].
> Searching there for '-fprofile' shows...
>
> LLVM_PROFDATA_FILE:PATH Path to a profdata file to pass into clang’s
> -fprofile-instr-use flag. This can only be specified if you’re
> building with clang.
>
> Unsure what to use!
>
> From my build-script (attached)...
>
> ##### BEGIN *** SECTION WILL BE DELETED ***
> #
> # XXX: TRYOUT #1: Use GOLD as linker
> # XXX: TRYOUT #2: Use '-O3' OptLevel
> # XXX: TRYOUT #3: Use #2 in combination with LTO and PGO ('-O3 -flto
> -fprofile-use')
> # XXX: TRYOUT #4: Use optimized tablegen
> #
> ### TRYOUT #1: GOLD <--- DONE
> # CMAKE_LINKER:FILEPATH=/usr/bin/ld
> # GOLD_EXECUTABLE:FILEPATH=/usr/bin/ld.gold
> # LLVM_TOOL_GOLD_BUILD:BOOL=ON
> #
> ### TRYOUT #2: OPTLEVEL '-O3' <--- NOP
> # CMAKE_ASM_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
> # CMAKE_CXX_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
> # CMAKE_C_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
> #
> ### TRYOUT #3: LTO AND PGO <--- UNSURE
> # LLVM_TOOL_LLVM_LTO_BUILD:BOOL=ON
> # LLVM_TOOL_LTO_BUILD:BOOL=ON
> # LLVM_USE_OPROFILE:BOOL=OFF
> #
> #### TRYOUT #4: TABLEGEN
> # LLVM_OPTIMIZED_TABLEGEN:BOOL=OFF
> #
> ##### END *** SECTION WILL BE DELETED ***
>
> Thanks for any help and ideas.
>
> Regards,
> - Sedat -
>
> [1] http://llvm.org/releases/3.8.0/docs/CMake.html

Attached a v2 of my build-script with "cmake options" pretty-fied.

- Sedat -
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build_llvm-toolchain_clang-cmake-ninja_v2.sh
Type: application/x-sh
Size: 4715 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160312/4b221c87/attachment.sh>