[llvm-dev] Building with LLVM_PARALLEL_XXX_JOBS

Sat Mar 12 03:45:26 PST 2016

On Fri, Mar 4, 2016 at 11:28 AM, Tilmann Scheller
<tilmann at osg.samsung.com> wrote:
> Hi Sedat,
>
> On 03/03/2016 08:09 AM, Sedat Dilek via llvm-dev wrote:
>>
>> It might be that a CLANG generated with LTO/PGO speeds up the build.
>> Can you confirm this?
>
> Yes, a Clang host compiler built with LTO or PGO is generally faster than an
> -O3 build.
>
> Some things to keep in mind when building the Clang host compiler:
>
> GCC:
>   - GCC 4.9 gives good results with PGO enabled (1.16x speedup over the -O3
> build), not so much with LTO (actually regresses performance over the -O3
> build, same for PGO vs PGO+LTO)
>   - GCC 5.1/5.2/5.3 can't build Clang with LTO enabled
> (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66027), that's supposed to be
> fixed in GCC 5.4
>
> Clang:
>   - PGO works and gives a good 1.12x speedup over the -O3 build (produced
> about 270GB of profiling data when I tried this in December last year, this
> should be addressed soon once the in-process profiling data merging lands)
>   - LTO provides a 1.03x speedup over the -O3 build
>   - I have not tried LTO+PGO with full Clang bootstrap profiling data but I
> would expect that it helps to increase the performance even further
>
>> Can you confirm binutils-gold speed up the build?
>
> Yes, gold is definitely faster than ld when building Clang/LLVM.
>
>> Has LLVM an own linker?
>> Can be used? Speedup the build?
>
> I haven't tried it but lld can definitely link Clang/LLVM on x86-64 Linux.
>
>> The blog-text mentioned to use optimized-tablegen.
>> Good? Bad? Ugly?
>
> Good, it helps to speed up debug builds.
>

[ CCed all folks who answered me ]

Hi,

I have built my llvm-toolchain v3.8.0 (FINAL) with binutils-gold v1.11
in a 1st run.

When building with cmake/ninja there are 150 "Linking" lines...

$ grep Linking logs/3.8.0_clang-3-8-0_cmake-3-4-3_ninja-1-6-0_gold-1-11_compile-jobs-2_link-jobs-1/build-log_llvm-toolchain-3.8.0.txt
| wc -l
150

I have the following cmake-options realized...

*** SNIP ***

### CMAKE OPTIONS
# NOTE #1: CMake Version 2.8.8 is the minimum required(Ubuntu/precise
ships v2.8.7 officially)
# NOTE #2: For fast builds use recommended CMake >= v3.2 (used:
v3.4.3) and Ninja (used: v1.6.0)
# NOTE #3: How to check available cmake-options?
# EXAMPLE #3: cd $BUILD_DIR ; cmake ../llvm -LA | egrep $CMAKE_OPTS
#
# CMake binary
CMAKE="cmake"
# CMake compiler options
COMPILERS_CMAKE_OPTS="-DCMAKE_C_COMPILER=$COMPILER_CC
-DCMAKE_CXX_COMPILER=$COMPILER_CXX"
# NOTE-1: cmake/ninja: Use LLVM_PARALLEL_COMPILE_JOBS and
LLVM_PARALLEL_LINK_JOBS options
# NOTE-2: For fast builds use available (online) CPUs +1 or set values
explicitly
# NOTE-3: For fast and safe linking use bintils-gold and LINK_JOBS="1"
COMPILE_JOBS="2"
##COMPILE_JOBS=$(($(getconf _NPROCESSORS_ONLN)+1))
LINK_JOBS="1"
JOBS_CMAKE_OPTS="-DLLVM_PARALLEL_COMPILE_JOBS=$COMPILE_JOBS
-DLLVM_PARALLEL_LINK_JOBS=$LINK_JOBS"
# Cmake linker options (here: Use binutils-gold to speedup build)
LINKER="/usr/bin/ld.gold"
CMAKE_LINKER="$LINKER"
CMAKE_LINKER_OPTS="-DCMAKE_LINKER=$CMAKE_LINKER"
# CMake Generators
CMAKE_GENERATORS="Ninja"
GENERATORS_CMAKE_OPTS="-G $CMAKE_GENERATORS"
# CMake configure settings
PREFIX_CMAKE_OPTS="-DCMAKE_INSTALL_PREFIX=$PREFIX"
OPTIMIZED_CMAKE_OPTS="-DCMAKE_BUILD_TYPE=RELEASE"
ASSERTIONS_CMAKE_OPTS="-DLLVM_ENABLE_ASSERTIONS=ON"
TARGETS_CMAKE_OPTS="-DLLVM_TARGETS_TO_BUILD=X86"
CONFIGURE_CMAKE_OPTS="$PREFIX_CMAKE_OPTS $OPTIMIZED_CMAKE_OPTS
$ASSERTIONS_CMAKE_OPTS $TARGETS_CMAKE_OPTS"
# All CMake options
CMAKE_OPTS="$COMPILERS_CMAKE_OPTS $JOBS_CMAKE_OPTS $CMAKE_LINKER_OPTS
$GENERATORS_CMAKE_OPTS $CONFIGURE_CMAKE_OPTS"

*** SNAP ***

Is LINK_JOBS="2" speeding up the build?
One guy of you told me to use "1" to be on the safe side - that is my default.

Personally, I do not think this is very very efficiently - more than
with binutild-bfd Linker.

Using llvm-link (in a 2nd build) - good|bad|ugly?

[ TODO#S: Before doing a 2nd build (and in a 3rd run using more
optimized binaries) ]

How do I anable LTO via CMAKE?
How do I enable PGO via CMAKE?

Grepping for 'lto' 'pgo' gives no help in [1].
Searching there for '-fprofile' shows...

LLVM_PROFDATA_FILE:PATH Path to a profdata file to pass into clang’s
-fprofile-instr-use flag. This can only be specified if you’re
building with clang.

Unsure what to use!

>From my build-script (attached)...

##### BEGIN *** SECTION WILL BE DELETED ***
#
# XXX: TRYOUT #1: Use GOLD as linker
# XXX: TRYOUT #2: Use '-O3' OptLevel
# XXX: TRYOUT #3: Use #2 in combination with LTO and PGO ('-O3 -flto
-fprofile-use')
# XXX: TRYOUT #4: Use optimized tablegen
#
### TRYOUT #1: GOLD <--- DONE
# CMAKE_LINKER:FILEPATH=/usr/bin/ld
# GOLD_EXECUTABLE:FILEPATH=/usr/bin/ld.gold
# LLVM_TOOL_GOLD_BUILD:BOOL=ON
#
### TRYOUT #2: OPTLEVEL '-O3' <--- NOP
# CMAKE_ASM_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
# CMAKE_CXX_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
# CMAKE_C_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
#
### TRYOUT #3: LTO AND PGO <--- UNSURE
# LLVM_TOOL_LLVM_LTO_BUILD:BOOL=ON
# LLVM_TOOL_LTO_BUILD:BOOL=ON
# LLVM_USE_OPROFILE:BOOL=OFF
#
#### TRYOUT #4: TABLEGEN
# LLVM_OPTIMIZED_TABLEGEN:BOOL=OFF
#
##### END *** SECTION WILL BE DELETED ***

Thanks for any help and ideas.

Regards,
- Sedat -

[1] http://llvm.org/releases/3.8.0/docs/CMake.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build_llvm-toolchain_clang-cmake-ninja.sh
Type: application/x-sh
Size: 4638 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160312/6249bc5d/attachment.sh>