[llvm-dev] Building with LLVM_PARALLEL_XXX_JOBS

Mon Mar 14 09:30:07 PDT 2016

> On Mar 12, 2016, at 3:45 AM, Sedat Dilek via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> On Fri, Mar 4, 2016 at 11:28 AM, Tilmann Scheller
> <tilmann at osg.samsung.com <mailto:tilmann at osg.samsung.com>> wrote:
>> Hi Sedat,
>> 
>> On 03/03/2016 08:09 AM, Sedat Dilek via llvm-dev wrote:
>>> 
>>> It might be that a CLANG generated with LTO/PGO speeds up the build.
>>> Can you confirm this?
>> 
>> Yes, a Clang host compiler built with LTO or PGO is generally faster than an
>> -O3 build.
>> 
>> Some things to keep in mind when building the Clang host compiler:
>> 
>> GCC:
>>  - GCC 4.9 gives good results with PGO enabled (1.16x speedup over the -O3
>> build), not so much with LTO (actually regresses performance over the -O3
>> build, same for PGO vs PGO+LTO)
>>  - GCC 5.1/5.2/5.3 can't build Clang with LTO enabled
>> (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66027), that's supposed to be
>> fixed in GCC 5.4
>> 
>> Clang:
>>  - PGO works and gives a good 1.12x speedup over the -O3 build (produced
>> about 270GB of profiling data when I tried this in December last year, this
>> should be addressed soon once the in-process profiling data merging lands)
>>  - LTO provides a 1.03x speedup over the -O3 build
>>  - I have not tried LTO+PGO with full Clang bootstrap profiling data but I
>> would expect that it helps to increase the performance even further
>> 
>>> Can you confirm binutils-gold speed up the build?
>> 
>> Yes, gold is definitely faster than ld when building Clang/LLVM.
>> 
>>> Has LLVM an own linker?
>>> Can be used? Speedup the build?
>> 
>> I haven't tried it but lld can definitely link Clang/LLVM on x86-64 Linux.
>> 
>>> The blog-text mentioned to use optimized-tablegen.
>>> Good? Bad? Ugly?
>> 
>> Good, it helps to speed up debug builds.
>> 
> 
> [ CCed all folks who answered me ]
> 
> Hi,
> 
> I have built my llvm-toolchain v3.8.0 (FINAL) with binutils-gold v1.11
> in a 1st run.
> 
> When building with cmake/ninja there are 150 "Linking" lines...
> 
> $ grep Linking logs/3.8.0_clang-3-8-0_cmake-3-4-3_ninja-1-6-0_gold-1-11_compile-jobs-2_link-jobs-1/build-log_llvm-toolchain-3.8.0.txt
> | wc -l
> 150
> 
> I have the following cmake-options realized...
> 
> *** SNIP ***
> 
> ### CMAKE OPTIONS
> # NOTE #1: CMake Version 2.8.8 is the minimum required(Ubuntu/precise
> ships v2.8.7 officially)
> # NOTE #2: For fast builds use recommended CMake >= v3.2 (used:
> v3.4.3) and Ninja (used: v1.6.0)
> # NOTE #3: How to check available cmake-options?
> # EXAMPLE #3: cd $BUILD_DIR ; cmake ../llvm -LA | egrep $CMAKE_OPTS
> #
> # CMake binary
> CMAKE="cmake"
> # CMake compiler options
> COMPILERS_CMAKE_OPTS="-DCMAKE_C_COMPILER=$COMPILER_CC
> -DCMAKE_CXX_COMPILER=$COMPILER_CXX"
> # NOTE-1: cmake/ninja: Use LLVM_PARALLEL_COMPILE_JOBS and
> LLVM_PARALLEL_LINK_JOBS options
> # NOTE-2: For fast builds use available (online) CPUs +1 or set values
> explicitly
> # NOTE-3: For fast and safe linking use bintils-gold and LINK_JOBS="1"
> COMPILE_JOBS="2"
> ##COMPILE_JOBS=$(($(getconf _NPROCESSORS_ONLN)+1))
> LINK_JOBS="1"
> JOBS_CMAKE_OPTS="-DLLVM_PARALLEL_COMPILE_JOBS=$COMPILE_JOBS
> -DLLVM_PARALLEL_LINK_JOBS=$LINK_JOBS"
> # Cmake linker options (here: Use binutils-gold to speedup build)
> LINKER="/usr/bin/ld.gold"
> CMAKE_LINKER="$LINKER"
> CMAKE_LINKER_OPTS="-DCMAKE_LINKER=$CMAKE_LINKER"
> # CMake Generators
> CMAKE_GENERATORS="Ninja"
> GENERATORS_CMAKE_OPTS="-G $CMAKE_GENERATORS"
> # CMake configure settings
> PREFIX_CMAKE_OPTS="-DCMAKE_INSTALL_PREFIX=$PREFIX"
> OPTIMIZED_CMAKE_OPTS="-DCMAKE_BUILD_TYPE=RELEASE"
> ASSERTIONS_CMAKE_OPTS="-DLLVM_ENABLE_ASSERTIONS=ON"
> TARGETS_CMAKE_OPTS="-DLLVM_TARGETS_TO_BUILD=X86"
> CONFIGURE_CMAKE_OPTS="$PREFIX_CMAKE_OPTS $OPTIMIZED_CMAKE_OPTS
> $ASSERTIONS_CMAKE_OPTS $TARGETS_CMAKE_OPTS"
> # All CMake options
> CMAKE_OPTS="$COMPILERS_CMAKE_OPTS $JOBS_CMAKE_OPTS $CMAKE_LINKER_OPTS
> $GENERATORS_CMAKE_OPTS $CONFIGURE_CMAKE_OPTS"
> 
> *** SNAP ***
> 
> Is LINK_JOBS="2" speeding up the build?
> One guy of you told me to use "1" to be on the safe side - that is my default.
> 
> Personally, I do not think this is very very efficiently - more than
> with binutild-bfd Linker.
> 
> Using llvm-link (in a 2nd build) - good|bad|ugly?
> 
> [ TODO#S: Before doing a 2nd build (and in a 3rd run using more
> optimized binaries) ]
> 
> How do I anable LTO via CMAKE?

LLVM_ENALBLE_LTO=On

> How do I enable PGO via CMAKE?

I’m actually in the middle of writing some documentation on this right now because I recently made this process a lot easier. Let me give you the high level overview.

The way PGO works is that you build an instrumented compiler, then you run the instrumented compiler against sample source files. While the instrumented compiler runs it will output a bunch of files containing performance counters (.profraw files). After generating all the profraw files you use llvm-profdata to merge the files into a single profdata file that you can feed into the LLVM_PROFDATA_FILE option.

Alas, there is an easier way to generate the profdata file!

If you look in the clang repo at <clang>/cmake/caches/README.txt you’ll see an explanation of how to use the PGO CMake cache file. The basic idea is you run:

$ cmake -G <generator> -C <path_to_clang>/cmake/caches/PGO.cmake <source dir>
$ <build tool> stage2-instrumented-generate-profdata

If you let that run for a few hours or so, it will place a profdata file in your build directory. This takes a really long time because it builds clang twice, and you *must* have compiler-rt in your build tree.

This process uses any source files under the perf-training directory as training data as long as the source files are marked up with LIT-style RUN lines.

After it finishes you can use “find . -name clang.profdata” to find it, but it should be at a path something like:

<build dir>/tools/clang/stage2-instrumented-bins/utils/perf-training/clang.profdata

You can feed that file into the LLVM_PROFDATA_FILE option when you build your optimized compiler.

-Chris

> 
> Grepping for 'lto' 'pgo' gives no help in [1].
> Searching there for '-fprofile' shows...
> 
> LLVM_PROFDATA_FILE:PATH Path to a profdata file to pass into clang’s
> -fprofile-instr-use flag. This can only be specified if you’re
> building with clang.
> 
> Unsure what to use!
> 
> From my build-script (attached)...
> 
> ##### BEGIN *** SECTION WILL BE DELETED ***
> #
> # XXX: TRYOUT #1: Use GOLD as linker
> # XXX: TRYOUT #2: Use '-O3' OptLevel
> # XXX: TRYOUT #3: Use #2 in combination with LTO and PGO ('-O3 -flto
> -fprofile-use')
> # XXX: TRYOUT #4: Use optimized tablegen
> #
> ### TRYOUT #1: GOLD <--- DONE
> # CMAKE_LINKER:FILEPATH=/usr/bin/ld
> # GOLD_EXECUTABLE:FILEPATH=/usr/bin/ld.gold
> # LLVM_TOOL_GOLD_BUILD:BOOL=ON
> #
> ### TRYOUT #2: OPTLEVEL '-O3' <--- NOP
> # CMAKE_ASM_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
> # CMAKE_CXX_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
> # CMAKE_C_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
> #
> ### TRYOUT #3: LTO AND PGO <--- UNSURE
> # LLVM_TOOL_LLVM_LTO_BUILD:BOOL=ON
> # LLVM_TOOL_LTO_BUILD:BOOL=ON
> # LLVM_USE_OPROFILE:BOOL=OFF
> #
> #### TRYOUT #4: TABLEGEN
> # LLVM_OPTIMIZED_TABLEGEN:BOOL=OFF
> #
> ##### END *** SECTION WILL BE DELETED ***
> 
> Thanks for any help and ideas.
> 
> Regards,
> - Sedat -
> 
> [1] http://llvm.org/releases/3.8.0/docs/CMake.html <http://llvm.org/releases/3.8.0/docs/CMake.html>
> <build_llvm-toolchain_clang-cmake-ninja.sh>_______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160314/9d130f7e/attachment-0001.html>