[llvm] r263834 - [Docs] New documentation for advanced build configurations
Chris Bieneman via llvm-commits
llvm-commits at lists.llvm.org
Fri Mar 18 15:05:16 PDT 2016
Thanks for the feedback.
Updated in r263840 & r263841.
-Chris
> On Mar 18, 2016, at 2:39 PM, Sean Silva <chisophugis at gmail.com> wrote:
>
> Nice!
>
> For consistency with other docs, can you rename this to AdvancedBuilds or AdvancedBuildConfigurations?
>
> A couple comments about non-ascii characters inline:
>
> On Fri, Mar 18, 2016 at 2:16 PM, Chris Bieneman via llvm-commits <llvm-commits at lists.llvm.org <mailto:llvm-commits at lists.llvm.org>> wrote:
> Author: cbieneman
> Date: Fri Mar 18 16:16:26 2016
> New Revision: 263834
>
> URL: http://llvm.org/viewvc/llvm-project?rev=263834&view=rev <http://llvm.org/viewvc/llvm-project?rev=263834&view=rev>
> Log:
> [Docs] New documentation for advanced build configurations
>
> This document covers how to use some of the new complex build configurations CMake supports.
>
> Feedback and improvements welcomed!
>
> Added:
> llvm/trunk/docs/Advanced_Builds.rst
> Modified:
> llvm/trunk/docs/CMake.rst
> llvm/trunk/docs/index.rst
>
> Added: llvm/trunk/docs/Advanced_Builds.rst
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/Advanced_Builds.rst?rev=263834&view=auto <http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/Advanced_Builds.rst?rev=263834&view=auto>
> ==============================================================================
> --- llvm/trunk/docs/Advanced_Builds.rst (added)
> +++ llvm/trunk/docs/Advanced_Builds.rst Fri Mar 18 16:16:26 2016
> @@ -0,0 +1,174 @@
> +=============================
> +Advanced Build Configurations
> +=============================
> +
> +.. contents::
> + :local:
> +
> +Introduction
> +============
> +
> +`CMake <http://www.cmake.org/ <http://www.cmake.org/>>`_ is a cross-platform build-generator tool. CMake
> +does not build the project, it generates the files needed by your build tool
> +(GNU make, Visual Studio, etc.) for building LLVM.
> +
> +If **you are a new contributor**, please start with the :doc:`GettingStarted` or
> +:doc:`CMake` pages. This page is intended for users doing more complex builds.
> +
> +Many of the examples below are written assuming specific CMake Generators.
> +Unless otherwise explicitly called out these commands should work with any CMake
> +generator.
> +
> +Bootstrap Builds
> +================
> +
> +The Clang CMake build system supports bootstrap (aka multi-stage) builds. At a
> +high level a multi-stage build is a chain of builds that pass data from one
> +stage into the next. The most common and simple version of this is a traditional
> +bootstrap build.
> +
> +In a simple two-stage bootstrap build, we build clang using the system compiler,
> +then use that just-built clang to build clang again. In CMake this simplest form
> +of a bootstrap build can be configured with a single option,
> +CLANG_ENABLE_BOOTSTRAP.
> +
> +.. code-block:: console
> +
> + $ make -G Ninja -DCLANG_ENABLE_BOOTSTRAP=On <path to source>
> + $ ninja stage2
> +
> +This command itself isn’t terribly useful because it assumes default
>
> Non-ascii
>
> +configurations for each stage. The next series of examples utilize CMake cache
> +scripts to provide more complex options.
> +
> +The clang build system refers to builds as stages. A stage1 build is a standard
> +build using the compiler installed on the host, and a stage2 build is built
> +using the stage1 compiler. This nomenclature holds up to more stages too. In
> +general a stage*n* build is built using the output from stage*n-1*.
> +
> +Apple Clang Builds (A More Complex Bootstrap)
> +=============================================
> +
> +Apple’s Clang builds are a slightly more complicated example of the simple
>
> Non-ascii
>
> +bootstrapping scenario. Apple Clang is built using a 2-stage build.
> +
> +The stage1 compiler is a host-only compiler with some options set. The stage1
> +compiler is a balance of optimization vs build time because it is a throwaway.
> +The stage2 compiler is the fully optimized compiler intended to ship to users.
> +
> +Setting up these compilers requires a lot of options. To simplify the
> +configuration the Apple Clang build settings are contained in CMake Cache files.
> +You can build an Apple Clang compiler using the following commands:
> +
> +.. code-block:: console
> +
> + $ make -G Ninja -C <path to clang>/cmake/caches/Apple-stage1.cmake <path to source>
> + $ ninja stage2-distribution
> +
> +This CMake invocation configures the stage1 host compiler, and sets
> +CLANG_BOOTSTRAP_CMAKE_ARGS to pass the Apple-stage2.cmake cache script to the
> +stage2 configuration step.
> +
> +When you build the stage2-distribution target it builds the minimal stage1
> +compiler and required tools, then configures and builds the stage2 compiler
> +based on the settings in Apple-stage2.cmake.
> +
> +This pattern of using cache scripts to set complex settings, and specifically to
> +make later stage builds include cache scripts is common in our more advanced
> +build configurations.
> +
> +Multi-stage PGO
> +===============
> +
> +Profile-Guided Optimizations (PGO) is a really great way to optimize the code
> +clang generates. Our multi-stage PGO builds are a workflow for generating PGO
> +profiles that can be used to optimize clang.
> +
> +At a high level, the way PGO works is that you build an instrumented compiler,
> +then you run the instrumented compiler against sample source files. While the
> +instrumented compiler runs it will output a bunch of files containing
> +performance counters (.profraw files). After generating all the profraw files
> +you use llvm-profdata to merge the files into a single profdata file that you
> +can feed into the LLVM_PROFDATA_FILE option.
> +
> +Our PGO.cmake cache script automates that whole process. You can use it by
> +running:
> +
> +.. code-block:: console
> +
> + $ make -G Ninja -C <path_to_clang>/cmake/caches/PGO.cmake <source dir>
> + $ ninja stage2-instrumented-generate-profdata
> +
> +If you let that run for a few hours or so, it will place a profdata file in your
> +build directory. This takes a really long time because it builds clang twice,
> +and you *must* have compiler-rt in your build tree.
> +
> +This process uses any source files under the perf-training directory as training
> +data as long as the source files are marked up with LIT-style RUN lines.
> +
> +After it finishes you can use “find . -name clang.profdata†to find it, but it
> +should be at a path something like:
> +
> +.. code-block:: console
> +
> + <build dir>/tools/clang/stage2-instrumented-bins/utils/perf-training/clang.profdata
> +
> +You can feed that file into the LLVM_PROFDATA_FILE option when you build your
> +optimized compiler.
> +
> +The PGO came cache has a slightly different stage naming scheme than other
> +multi-stage builds. It generates three stages; stage1, stage2-instrumented, and
> +stage2. Both of the stage2 builds are built using the stage1 compiler.
> +
> +The PGO came cache generates the following additional targets:
> +
> +**stage2-instrumented**
> + Builds a stage1 x86 compiler, runtime, and required tools (llvm-config,
> + llvm-profdata) then uses that compiler to build an instrumented stage2 compiler.
> +
> +**stage2-instrumented-generate-profdata**
> + Depends on “stage2-instrumented†and will use the instrumented compiler to
> + generate profdata based on the training files in <clang>/utils/perf-training
> +
> +**stage2**
> + Depends of “stage2-instrumented-generate-profdata†and will use the stage1
>
> Non-ascii
>
> -- Sean Silva
>
> + compiler with the stage2 profdata to build a PGO-optimized compiler.
> +
> +**stage2-check-llvm**
> + Depends on stage2 and runs check-llvm using the stage2 compiler.
> +
> +**stage2-check-clang**
> + Depends on stage2 and runs check-clang using the stage2 compiler.
> +
> +**stage2-check-all**
> + Depends on stage2 and runs check-all using the stage2 compiler.
> +
> +**stage2-test-suite**
> + Depends on stage2 and runs the test-suite using the stage3 compiler (requires
> + in-tree test-suite).
> +
> +3-Stage Non-Determinism
> +=======================
> +
> +In the ancient lore of compilers non-determinism is like the multi-headed hydra.
> +Whenever it's head pops up, terror and chaos ensue.
> +
> +Historically one of the tests to verify that a compiler was deterministic would
> +be a three stage build. The idea of a three stage build is you take your sources
> +and build a compiler (stage1), then use that compiler to rebuild the sources
> +(stage2), then you use that compiler to rebuild the sources a third time
> +(stage3) with an identical configuration to the stage2 build. At the end of
> +this, you have a stage2 and stage3 compiler that should be bit-for-bit
> +identical.
> +
> +You can perform one of these 3-stage builds with LLVM & clang using the
> +following commands:
> +
> +.. code-block:: console
> +
> + $ cmake -G Ninja -C <path_to_clang>/cmake/caches/3-stage.cmake <source dir>
> + $ ninja stage3
> +
> +After the build you can compare the stage2 & stage3 compilers. We have a bot
> +setup `here <http://lab.llvm.org:8011/builders/clang-3stage-ubuntu <http://lab.llvm.org:8011/builders/clang-3stage-ubuntu>>`_ that runs
> +this build and compare configuration.
>
> Modified: llvm/trunk/docs/CMake.rst
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/CMake.rst?rev=263834&r1=263833&r2=263834&view=diff <http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/CMake.rst?rev=263834&r1=263833&r2=263834&view=diff>
> ==============================================================================
> --- llvm/trunk/docs/CMake.rst (original)
> +++ llvm/trunk/docs/CMake.rst Fri Mar 18 16:16:26 2016
> @@ -474,6 +474,41 @@ LLVM-specific variables
> If you want to build LLVM as a shared library, you should use the
> ``LLVM_BUILD_LLVM_DYLIB`` option.
>
> +CMake Caches
> +============
> +
> +Recently LLVM and Clang have been adding some more complicated build system
> +features. Utilizing these new features often involves a complicated chain of
> +CMake variables passed on the command line. Clang provides a collection of CMake
> +cache scripts to make these features more approachable.
> +
> +CMake cache files are utilized using CMake's -C flag:
> +
> +.. code-block:: console
> +
> + $ cmake -C <path to cache file> <path to sources>
> +
> +CMake cache scripts are processed in an isolated scope, only cached variables
> +remain set when the main configuration runs. CMake cached variables do not reset
> +variables that are already set unless the FORCE option is specified.
> +
> +A few notes about CMake Caches:
> +
> +- Order of command line arguments is important
> +
> + - -D arguments specified before -C are set before the cache is processed and
> + can be read inside the cache file
> + - -D arguments specified after -C are set after the cache is processed and
> + are unset inside the cache file
> +
> +- All -D arguments will override cache file settings
> +- CMAKE_TOOLCHAIN_FILE is evaluated after both the cache file and the command
> + line arguments
> +- It is recommended that all -D options should be specified *before* -C
> +
> +For more information about some of the advanced build configurations supported
> +via Cache files see :doc:`Advanced_Builds`.
> +
> Executing the test suite
> ========================
>
>
> Modified: llvm/trunk/docs/index.rst
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/index.rst?rev=263834&r1=263833&r2=263834&view=diff <http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/index.rst?rev=263834&r1=263833&r2=263834&view=diff>
> ==============================================================================
> --- llvm/trunk/docs/index.rst (original)
> +++ llvm/trunk/docs/index.rst Fri Mar 18 16:16:26 2016
> @@ -65,6 +65,7 @@ representation.
> :hidden:
>
> CMake
> + Advanced_Builds
> HowToBuildOnARM
> HowToCrossCompileLLVM
> CommandGuide/index
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org <mailto:llvm-commits at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20160318/e04ae6c7/attachment.html>
More information about the llvm-commits
mailing list