[llvm] r263834 - [Docs] New documentation for advanced build configurations

[Sean Silva via llvm-commits]

Nice!

For consistency with other docs, can you rename this to AdvancedBuilds or
AdvancedBuildConfigurations?

A couple comments about non-ascii characters inline:

On Fri, Mar 18, 2016 at 2:16 PM, Chris Bieneman via llvm-commits <
llvm-commits at lists.llvm.org> wrote:

> Author: cbieneman
> Date: Fri Mar 18 16:16:26 2016
> New Revision: 263834
>
> URL: http://llvm.org/viewvc/llvm-project?rev=263834&view=rev
> Log:
> [Docs] New documentation for advanced build configurations
>
> This document covers how to use some of the new complex build
> configurations CMake supports.
>
> Feedback and improvements welcomed!
>
> Added:
>     llvm/trunk/docs/Advanced_Builds.rst
> Modified:
>     llvm/trunk/docs/CMake.rst
>     llvm/trunk/docs/index.rst
>
> Added: llvm/trunk/docs/Advanced_Builds.rst
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/Advanced_Builds.rst?rev=263834&view=auto
>
> ==============================================================================
> --- llvm/trunk/docs/Advanced_Builds.rst (added)
> +++ llvm/trunk/docs/Advanced_Builds.rst Fri Mar 18 16:16:26 2016
> @@ -0,0 +1,174 @@
> +=============================
> +Advanced Build Configurations
> +=============================
> +
> +.. contents::
> +   :local:
> +
> +Introduction
> +============
> +
> +`CMake <http://www.cmake.org/>`_ is a cross-platform build-generator
> tool. CMake
> +does not build the project, it generates the files needed by your build
> tool
> +(GNU make, Visual Studio, etc.) for building LLVM.
> +
> +If **you are a new contributor**, please start with the
> :doc:`GettingStarted` or
> +:doc:`CMake` pages. This page is intended for users doing more complex
> builds.
> +
> +Many of the examples below are written assuming specific CMake Generators.
> +Unless otherwise explicitly called out these commands should work with
> any CMake
> +generator.
> +
> +Bootstrap Builds
> +================
> +
> +The Clang CMake build system supports bootstrap (aka multi-stage) builds.
> At a
> +high level a multi-stage build is a chain of builds that pass data from
> one
> +stage into the next. The most common and simple version of this is a
> traditional
> +bootstrap build.
> +
> +In a simple two-stage bootstrap build, we build clang using the system
> compiler,
> +then use that just-built clang to build clang again. In CMake this
> simplest form
> +of a bootstrap build can be configured with a single option,
> +CLANG_ENABLE_BOOTSTRAP.
> +
> +.. code-block:: console
> +
> +  $ make -G Ninja -DCLANG_ENABLE_BOOTSTRAP=On <path to source>
> +  $ ninja stage2
> +
> +This command itself isn’t terribly useful because it assumes default
>

Non-ascii


> +configurations for each stage. The next series of examples utilize CMake
> cache
> +scripts to provide more complex options.
> +
> +The clang build system refers to builds as stages. A stage1 build is a
> standard
> +build using the compiler installed on the host, and a stage2 build is
> built
> +using the stage1 compiler. This nomenclature holds up to more stages too.
> In
> +general a stage*n* build is built using the output from stage*n-1*.
> +
> +Apple Clang Builds (A More Complex Bootstrap)
> +=============================================
> +
> +Apple’s Clang builds are a slightly more complicated example of the
> simple
>

Non-ascii


> +bootstrapping scenario. Apple Clang is built using a 2-stage build.
> +
> +The stage1 compiler is a host-only compiler with some options set. The
> stage1
> +compiler is a balance of optimization vs build time because it is a
> throwaway.
> +The stage2 compiler is the fully optimized compiler intended to ship to
> users.
> +
> +Setting up these compilers requires a lot of options. To simplify the
> +configuration the Apple Clang build settings are contained in CMake Cache
> files.
> +You can build an Apple Clang compiler using the following commands:
> +
> +.. code-block:: console
> +
> +  $ make -G Ninja -C <path to clang>/cmake/caches/Apple-stage1.cmake
> <path to source>
> +  $ ninja stage2-distribution
> +
> +This CMake invocation configures the stage1 host compiler, and sets
> +CLANG_BOOTSTRAP_CMAKE_ARGS to pass the Apple-stage2.cmake cache script to
> the
> +stage2 configuration step.
> +
> +When you build the stage2-distribution target it builds the minimal stage1
> +compiler and required tools, then configures and builds the stage2
> compiler
> +based on the settings in Apple-stage2.cmake.
> +
> +This pattern of using cache scripts to set complex settings, and
> specifically to
> +make later stage builds include cache scripts is common in our more
> advanced
> +build configurations.
> +
> +Multi-stage PGO
> +===============
> +
> +Profile-Guided Optimizations (PGO) is a really great way to optimize the
> code
> +clang generates. Our multi-stage PGO builds are a workflow for generating
> PGO
> +profiles that can be used to optimize clang.
> +
> +At a high level, the way PGO works is that you build an instrumented
> compiler,
> +then you run the instrumented compiler against sample source files. While
> the
> +instrumented compiler runs it will output a bunch of files containing
> +performance counters (.profraw files). After generating all the profraw
> files
> +you use llvm-profdata to merge the files into a single profdata file that
> you
> +can feed into the LLVM_PROFDATA_FILE option.
> +
> +Our PGO.cmake cache script automates that whole process. You can use it by
> +running:
> +
> +.. code-block:: console
> +
> +  $ make -G Ninja -C <path_to_clang>/cmake/caches/PGO.cmake <source dir>
> +  $ ninja stage2-instrumented-generate-profdata
> +
> +If you let that run for a few hours or so, it will place a profdata file
> in your
> +build directory. This takes a really long time because it builds clang
> twice,
> +and you *must* have compiler-rt in your build tree.
> +
> +This process uses any source files under the perf-training directory as
> training
> +data as long as the source files are marked up with LIT-style RUN lines.
> +
> +After it finishes you can use “find . -name clang.profdata† to find
> it, but it
> +should be at a path something like:
> +
> +.. code-block:: console
> +
> +  <build
> dir>/tools/clang/stage2-instrumented-bins/utils/perf-training/clang.profdata
> +
> +You can feed that file into the LLVM_PROFDATA_FILE option when you build
> your
> +optimized compiler.
> +
> +The PGO came cache has a slightly different stage naming scheme than other
> +multi-stage builds. It generates three stages; stage1,
> stage2-instrumented, and
> +stage2. Both of the stage2 builds are built using the stage1 compiler.
> +
> +The PGO came cache generates the following additional targets:
> +
> +**stage2-instrumented**
> +  Builds a stage1 x86 compiler, runtime, and required tools (llvm-config,
> +  llvm-profdata) then uses that compiler to build an instrumented stage2
> compiler.
> +
> +**stage2-instrumented-generate-profdata**
> +  Depends on “stage2-instrumented† and will use the instrumented
> compiler to
> +  generate profdata based on the training files in
> <clang>/utils/perf-training
> +
> +**stage2**
> +  Depends of “stage2-instrumented-generate-profdata† and will use the
> stage1
>

Non-ascii

-- Sean Silva


> +  compiler with the stage2 profdata to build a PGO-optimized compiler.
> +
> +**stage2-check-llvm**
> +  Depends on stage2 and runs check-llvm using the stage2 compiler.
> +
> +**stage2-check-clang**
> +  Depends on stage2 and runs check-clang using the stage2 compiler.
> +
> +**stage2-check-all**
> +  Depends on stage2 and runs check-all using the stage2 compiler.
> +
> +**stage2-test-suite**
> +  Depends on stage2 and runs the test-suite using the stage3 compiler
> (requires
> +  in-tree test-suite).
> +
> +3-Stage Non-Determinism
> +=======================
> +
> +In the ancient lore of compilers non-determinism is like the multi-headed
> hydra.
> +Whenever it's head pops up, terror and chaos ensue.
> +
> +Historically one of the tests to verify that a compiler was deterministic
> would
> +be a three stage build. The idea of a three stage build is you take your
> sources
> +and build a compiler (stage1), then use that compiler to rebuild the
> sources
> +(stage2), then you use that compiler to rebuild the sources a third time
> +(stage3) with an identical configuration to the stage2 build. At the end
> of
> +this, you have a stage2 and stage3 compiler that should be bit-for-bit
> +identical.
> +
> +You can perform one of these 3-stage builds with LLVM & clang using the
> +following commands:
> +
> +.. code-block:: console
> +
> +  $ cmake -G Ninja -C <path_to_clang>/cmake/caches/3-stage.cmake <source
> dir>
> +  $ ninja stage3
> +
> +After the build you can compare the stage2 & stage3 compilers. We have a
> bot
> +setup `here <http://lab.llvm.org:8011/builders/clang-3stage-ubuntu>`_
> that runs
> +this build and compare configuration.
>
> Modified: llvm/trunk/docs/CMake.rst
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/CMake.rst?rev=263834&r1=263833&r2=263834&view=diff
>
> ==============================================================================
> --- llvm/trunk/docs/CMake.rst (original)
> +++ llvm/trunk/docs/CMake.rst Fri Mar 18 16:16:26 2016
> @@ -474,6 +474,41 @@ LLVM-specific variables
>              If you want to build LLVM as a shared library, you should use
> the
>              ``LLVM_BUILD_LLVM_DYLIB`` option.
>
> +CMake Caches
> +============
> +
> +Recently LLVM and Clang have been adding some more complicated build
> system
> +features. Utilizing these new features often involves a complicated chain
> of
> +CMake variables passed on the command line. Clang provides a collection
> of CMake
> +cache scripts to make these features more approachable.
> +
> +CMake cache files are utilized using CMake's -C flag:
> +
> +.. code-block:: console
> +
> +  $ cmake -C <path to cache file> <path to sources>
> +
> +CMake cache scripts are processed in an isolated scope, only cached
> variables
> +remain set when the main configuration runs. CMake cached variables do
> not reset
> +variables that are already set unless the FORCE option is specified.
> +
> +A few notes about CMake Caches:
> +
> +- Order of command line arguments is important
> +
> +  - -D arguments specified before -C are set before the cache is
> processed and
> +    can be read inside the cache file
> +  - -D arguments specified after -C are set after the cache is processed
> and
> +    are unset inside the cache file
> +
> +- All -D arguments will override cache file settings
> +- CMAKE_TOOLCHAIN_FILE is evaluated after both the cache file and the
> command
> +  line arguments
> +- It is recommended that all -D options should be specified *before* -C
> +
> +For more information about some of the advanced build configurations
> supported
> +via Cache files see :doc:`Advanced_Builds`.
> +
>  Executing the test suite
>  ========================
>
>
> Modified: llvm/trunk/docs/index.rst
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/index.rst?rev=263834&r1=263833&r2=263834&view=diff
>
> ==============================================================================
> --- llvm/trunk/docs/index.rst (original)
> +++ llvm/trunk/docs/index.rst Fri Mar 18 16:16:26 2016
> @@ -65,6 +65,7 @@ representation.
>     :hidden:
>
>     CMake
> +   Advanced_Builds
>     HowToBuildOnARM
>     HowToCrossCompileLLVM
>     CommandGuide/index
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20160318/f3b47d82/attachment.html>