[llvm] r263834 - [Docs] New documentation for advanced build configurations

[Chris Bieneman via llvm-commits]

Author: cbieneman
Date: Fri Mar 18 16:16:26 2016
New Revision: 263834

URL: http://llvm.org/viewvc/llvm-project?rev=263834&view=rev
Log:
[Docs] New documentation for advanced build configurations

This document covers how to use some of the new complex build configurations CMake supports.

Feedback and improvements welcomed!

Added:
    llvm/trunk/docs/Advanced_Builds.rst
Modified:
    llvm/trunk/docs/CMake.rst
    llvm/trunk/docs/index.rst

Added: llvm/trunk/docs/Advanced_Builds.rst
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/Advanced_Builds.rst?rev=263834&view=auto
==============================================================================
--- llvm/trunk/docs/Advanced_Builds.rst (added)
+++ llvm/trunk/docs/Advanced_Builds.rst Fri Mar 18 16:16:26 2016
@@ -0,0 +1,174 @@
+=============================
+Advanced Build Configurations
+=============================
+
+.. contents::
+   :local:
+
+Introduction
+============
+
+`CMake <http://www.cmake.org/>`_ is a cross-platform build-generator tool. CMake
+does not build the project, it generates the files needed by your build tool
+(GNU make, Visual Studio, etc.) for building LLVM.
+
+If **you are a new contributor**, please start with the :doc:`GettingStarted` or
+:doc:`CMake` pages. This page is intended for users doing more complex builds.
+
+Many of the examples below are written assuming specific CMake Generators.
+Unless otherwise explicitly called out these commands should work with any CMake
+generator.
+
+Bootstrap Builds
+================
+
+The Clang CMake build system supports bootstrap (aka multi-stage) builds. At a
+high level a multi-stage build is a chain of builds that pass data from one
+stage into the next. The most common and simple version of this is a traditional
+bootstrap build.
+
+In a simple two-stage bootstrap build, we build clang using the system compiler,
+then use that just-built clang to build clang again. In CMake this simplest form
+of a bootstrap build can be configured with a single option,
+CLANG_ENABLE_BOOTSTRAP.
+
+.. code-block:: console
+
+  $ make -G Ninja -DCLANG_ENABLE_BOOTSTRAP=On <path to source>
+  $ ninja stage2
+
+This command itself isn’t terribly useful because it assumes default
+configurations for each stage. The next series of examples utilize CMake cache
+scripts to provide more complex options.
+
+The clang build system refers to builds as stages. A stage1 build is a standard
+build using the compiler installed on the host, and a stage2 build is built
+using the stage1 compiler. This nomenclature holds up to more stages too. In
+general a stage*n* build is built using the output from stage*n-1*.
+
+Apple Clang Builds (A More Complex Bootstrap)
+=============================================
+
+Apple’s Clang builds are a slightly more complicated example of the simple
+bootstrapping scenario. Apple Clang is built using a 2-stage build.
+
+The stage1 compiler is a host-only compiler with some options set. The stage1
+compiler is a balance of optimization vs build time because it is a throwaway.
+The stage2 compiler is the fully optimized compiler intended to ship to users.
+
+Setting up these compilers requires a lot of options. To simplify the
+configuration the Apple Clang build settings are contained in CMake Cache files.
+You can build an Apple Clang compiler using the following commands:
+
+.. code-block:: console
+
+  $ make -G Ninja -C <path to clang>/cmake/caches/Apple-stage1.cmake <path to source>
+  $ ninja stage2-distribution
+
+This CMake invocation configures the stage1 host compiler, and sets
+CLANG_BOOTSTRAP_CMAKE_ARGS to pass the Apple-stage2.cmake cache script to the
+stage2 configuration step.
+
+When you build the stage2-distribution target it builds the minimal stage1
+compiler and required tools, then configures and builds the stage2 compiler
+based on the settings in Apple-stage2.cmake.
+
+This pattern of using cache scripts to set complex settings, and specifically to
+make later stage builds include cache scripts is common in our more advanced
+build configurations.
+
+Multi-stage PGO
+===============
+
+Profile-Guided Optimizations (PGO) is a really great way to optimize the code
+clang generates. Our multi-stage PGO builds are a workflow for generating PGO
+profiles that can be used to optimize clang.
+
+At a high level, the way PGO works is that you build an instrumented compiler,
+then you run the instrumented compiler against sample source files. While the
+instrumented compiler runs it will output a bunch of files containing
+performance counters (.profraw files). After generating all the profraw files
+you use llvm-profdata to merge the files into a single profdata file that you
+can feed into the LLVM_PROFDATA_FILE option.
+
+Our PGO.cmake cache script automates that whole process. You can use it by
+running:
+
+.. code-block:: console
+
+  $ make -G Ninja -C <path_to_clang>/cmake/caches/PGO.cmake <source dir>
+  $ ninja stage2-instrumented-generate-profdata
+
+If you let that run for a few hours or so, it will place a profdata file in your
+build directory. This takes a really long time because it builds clang twice,
+and you *must* have compiler-rt in your build tree.
+
+This process uses any source files under the perf-training directory as training
+data as long as the source files are marked up with LIT-style RUN lines.
+
+After it finishes you can use “find . -name clang.profdata” to find it, but it
+should be at a path something like:
+
+.. code-block:: console
+
+  <build dir>/tools/clang/stage2-instrumented-bins/utils/perf-training/clang.profdata
+
+You can feed that file into the LLVM_PROFDATA_FILE option when you build your
+optimized compiler.
+
+The PGO came cache has a slightly different stage naming scheme than other
+multi-stage builds. It generates three stages; stage1, stage2-instrumented, and
+stage2. Both of the stage2 builds are built using the stage1 compiler.
+
+The PGO came cache generates the following additional targets:
+
+**stage2-instrumented**
+  Builds a stage1 x86 compiler, runtime, and required tools (llvm-config,
+  llvm-profdata) then uses that compiler to build an instrumented stage2 compiler.
+
+**stage2-instrumented-generate-profdata**
+  Depends on “stage2-instrumented” and will use the instrumented compiler to
+  generate profdata based on the training files in <clang>/utils/perf-training
+
+**stage2**
+  Depends of “stage2-instrumented-generate-profdata” and will use the stage1
+  compiler with the stage2 profdata to build a PGO-optimized compiler.
+
+**stage2-check-llvm**
+  Depends on stage2 and runs check-llvm using the stage2 compiler.
+
+**stage2-check-clang**
+  Depends on stage2 and runs check-clang using the stage2 compiler.
+
+**stage2-check-all**
+  Depends on stage2 and runs check-all using the stage2 compiler.
+
+**stage2-test-suite**
+  Depends on stage2 and runs the test-suite using the stage3 compiler (requires
+  in-tree test-suite).
+
+3-Stage Non-Determinism
+=======================
+
+In the ancient lore of compilers non-determinism is like the multi-headed hydra.
+Whenever it's head pops up, terror and chaos ensue.
+
+Historically one of the tests to verify that a compiler was deterministic would
+be a three stage build. The idea of a three stage build is you take your sources
+and build a compiler (stage1), then use that compiler to rebuild the sources
+(stage2), then you use that compiler to rebuild the sources a third time
+(stage3) with an identical configuration to the stage2 build. At the end of
+this, you have a stage2 and stage3 compiler that should be bit-for-bit
+identical.
+
+You can perform one of these 3-stage builds with LLVM & clang using the
+following commands:
+
+.. code-block:: console
+
+  $ cmake -G Ninja -C <path_to_clang>/cmake/caches/3-stage.cmake <source dir>
+  $ ninja stage3
+
+After the build you can compare the stage2 & stage3 compilers. We have a bot
+setup `here <http://lab.llvm.org:8011/builders/clang-3stage-ubuntu>`_ that runs
+this build and compare configuration.

Modified: llvm/trunk/docs/CMake.rst
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/CMake.rst?rev=263834&r1=263833&r2=263834&view=diff
==============================================================================
--- llvm/trunk/docs/CMake.rst (original)
+++ llvm/trunk/docs/CMake.rst Fri Mar 18 16:16:26 2016
@@ -474,6 +474,41 @@ LLVM-specific variables
             If you want to build LLVM as a shared library, you should use the
             ``LLVM_BUILD_LLVM_DYLIB`` option.
 
+CMake Caches
+============
+
+Recently LLVM and Clang have been adding some more complicated build system
+features. Utilizing these new features often involves a complicated chain of
+CMake variables passed on the command line. Clang provides a collection of CMake
+cache scripts to make these features more approachable.
+
+CMake cache files are utilized using CMake's -C flag:
+
+.. code-block:: console
+
+  $ cmake -C <path to cache file> <path to sources>
+
+CMake cache scripts are processed in an isolated scope, only cached variables
+remain set when the main configuration runs. CMake cached variables do not reset
+variables that are already set unless the FORCE option is specified.
+
+A few notes about CMake Caches:
+
+- Order of command line arguments is important
+
+  - -D arguments specified before -C are set before the cache is processed and
+    can be read inside the cache file
+  - -D arguments specified after -C are set after the cache is processed and
+    are unset inside the cache file
+
+- All -D arguments will override cache file settings
+- CMAKE_TOOLCHAIN_FILE is evaluated after both the cache file and the command
+  line arguments
+- It is recommended that all -D options should be specified *before* -C
+
+For more information about some of the advanced build configurations supported
+via Cache files see :doc:`Advanced_Builds`.
+
 Executing the test suite
 ========================
 

Modified: llvm/trunk/docs/index.rst
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/index.rst?rev=263834&r1=263833&r2=263834&view=diff
==============================================================================
--- llvm/trunk/docs/index.rst (original)
+++ llvm/trunk/docs/index.rst Fri Mar 18 16:16:26 2016
@@ -65,6 +65,7 @@ representation.
    :hidden:
 
    CMake
+   Advanced_Builds
    HowToBuildOnARM
    HowToCrossCompileLLVM
    CommandGuide/index