[PATCH] [DOCS] How to Setup a Windows Builder
Mikael Lyngvig
mikael at lyngvig.org
Mon Nov 11 15:50:15 PST 2013
This document describes how to set up a Windows builder and is a document
that quite a few have asked about during the past few years.
Please find attached diff.
Please direct all criticism, error reports, and enhancement requests to me
and I'll see what I can do.
Sincerely,
Mikael Lyngvig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20131112/f20302a0/attachment.html>
-------------- next part --------------
Index: docs/HowToSetupAWindowsBuilder.rst
===================================================================
--- docs/HowToSetupAWindowsBuilder.rst (revision 0)
+++ docs/HowToSetupAWindowsBuilder.rst (revision 0)
@@ -0,0 +1,606 @@
+=========================================================
+How to Setup Windows for the LLVM Buildbot Infrastructure
+=========================================================
+
+.. contents::
+ :local:
+ :depth: 1
+
+
+Introduction
+============
+Building LLVM on Windows using the MinGW32, MinGW64, and Visual Studio toolchains
+is complicated. This document aims to make it easy for you to configure
+Windows to build LLVM with the major windows toolchains, bourne shells, and
+natively with Microsoft's ``cmd.exe``.
+
+In the text that follows, your LLVM working directory will be referred to as
+``%LLVMDIR%``.
+
+MinGW, MSYS, and Cygwin for Beginners
+-------------------------------------
+MinGW stands for *Minimalistic Gnu for Windows*. The intent of the MinGW
+project is to deliver a set of ready-to-use ports of GNU software on Windows.
+
+MinGW currently exists in two versions: MinGW32 (32-bit tools) and MinGW64
+(64-bit tools). The 32-bit tools are readily available from the `main MinGW
+website <http://www.mingw.org>`_. The 64-bit tools are available from the
+`MinGW W64 website <http://mingw-w64.sourceforge.net>`_.
+
+MinGW tools are distributed as standalone packages, or packaged with Cygwin.
+The Cygwin setup tool is a GUI package manager with the MinGW package names
+prefixed with mingw32- or mingw64-.
+Cygwin provides a POSIX runtime environment for POSIX applications ported to
+Windows. In this example, Cygwin will be used to supply missing utilities.
+
+We will primarily use the standalone MinGW tools for the GCC builds of LLVM.
+
+The standalone MinGW tools come with an alternative to Microsoft's ``cmd.exe``
+called MSYS. MSYS stands for "Minimal SYStem", a Bourne Shell command line
+interpreter and utilities that are suited for porting Open Source applications
+to MS-Windows.
+
+**Notice:** You don't need Microsoft Visual Studio to build LLVM and Clang on
+Windows. If your application needs to link in special Microsoft libraries,
+that are not part of the MinGW distribution, you can get access to those by
+downloading the *Microsoft Windows SDK for Windows 7 and .NET 4.0* from the
+`Microsoft Download Center
+<http://www.microsoft.com/en-us/download/details.aspx?id=8442>`_.
+
+
+System Requirements
+===================
+If you are going to debug LLVM or Clang, you need at least 6 gigabytes of
+memory. This is because the GNU linker (``ld``) eats a lot of memory when
+linking in libraries that contain debug information. If you are not going
+to debug LLVM and/or Clang, you can do with 4 gigabytes.
+
+You'll need about 5 gigabytes of free disk space for the tools and the build.
+
+For use with Buildbot, create a new user account named ``buildbot``. Give it
+any password you like, but be sure to not use a password that you already use
+on your network so as to make sure that hackers and hostile build masters
+cannot access all of your computers on your local network.
+
+Mark the user ``buildbot`` for automatic login. On Windows 7, you do this by
+running the command ``netplwiz`` and then uncheck the *Users must enter a user
+name and password to use this computer* field. Then click the Apply button and
+enter the user name ``buildbot`` and the password that you gave this account.
+
+Software Setup
+==============
+In this chapter, you'll be walked through downloading and installing all of
+the software packages that are needed to do a full debug or release build
+and test of LLVM.
+
+
+Installing MinGW 32-bit
+-----------------------
+Download and install MinGW32 using the MinGW installer found at the `MinGW
+project at SourceForge <http://sourceforge.net/projects/mingw/>`_. Download
+the file marked as the default download (``mingw-get-inst-YYYYMMDD.exe``).
+Run it then select ``Download latest repository catalogues``. Install
+to ``C:\Mingw32`` as you'll perhaps also be installing MinGW64 later on.
+Select and check the ``C Compiler`` and the ``C++ Compiler``.
+
+**Notice:** If you only want to build 64-bit programs, you do **not** need to
+install MinGW32.
+
+
+Installing MinGW 64-bit
+-----------------------
+Install MinGW64 by downloading the latest version from the
+`Download MinGW-w64 <http://sourceforge.net/projects/mingw-w64/files/latest/download>`_.
+``mingw-w64-<version>.tar.bz2``. Untar it, using 7-Zip from
+`www.7-Zip.org <http://www.7-zip.org/>`_, to ``C:\MinGW64``,
+or Cygwin with the bzip2 package installed.
+
+**Notice:** If you only want to build 32-bit programs, you do **not** need to
+install MinGW64.
+
+Installing GnuWin32
+-------------------
+Install GnuWin32 by following the *Looking for the latest version?* link at
+`GnuWin32 on SourceForge.net <gnuwin32.sourceforge.net>`_. Download and run
+the executable. Once it has run to completion, you need to run both the
+download and install scripts that are in the download directory
+(please notice that you should specify an installation directory, say,
+C:\\GnuWin32, to the install script). The download directory is by default
+identical to C:\Users\Build\Downloads\GetGnuWin32.
+
+The primary purpose of GnuWin32 is to supply utilities missing in MinGW-32
+MSYS. GnuWin32 will be after MSYS in the path and handle utilities that are not
+available in MSYS.
+
+**Notice:** If you do not plan to run the test suite, or debug the code,
+you don't need GnuWin32. You can build LLVM + Clang with only Subversion,
+MingwNN, and CMake.
+
+Installing Subversion
+---------------------
+Download and install Subversion from the `Subversion download page
+<http://tortoisesvn.net/downloads.html>`_. Pick the appropriate
+version for your OS. Install it using the ``Typical`` choice.
+
+**Notice:** You will not need Subversion if you are building from an official
+release. However, the recommended path is to build from the Subversion trunk
+as that gives you the most recent code and because you can that way much
+easier integrate new features and code changes into your own project.
+
+
+Installing CMake
+----------------
+Navigate to the `CMake download page
+<http://www.cmake.org/cmake/resources/software.html#latest>`_ and download
+the latest Win32 installer. Run it and install CMake to its default location
+while making sure you select the ``Add CMake to the system PATH for current
+user`` option during setup.
+
+
+Installing Ninja
+----------------
+Ninja is a *very* fast build tool that is one of the builders in CMake.
+You can download a binary from
+`CMakeScript <http://sourceforge.net/projects/cmakescript/files/>`_. You
+can save it to your standard tools directory or to ``%LLVMDIR%`` and invoke it
+from there.
+
+**Notice:** This version of Ninja requires you to have the Microsoft Visual
+C++ 2010 Redistributable installed. You can get it from `Microsoft's Download
+Center <http://www.microsoft.com/en-us/download/details.aspx?id=5555>`_.
+
+
+Installing Python v2.x
+----------------------
+Download and install the latest release of v2.x of Python from the
+`Python website <http://www.python.org/download/>`_.
+
+Installing Python v2.x SetupTools
+---------------------------------
+Download and install the latest version of SetupTools for Python from the PyPi
+page for the SetupTools. Be sure to pick the version that matches your Python
+version!
+
+Installing the Python Win32 Bindings
+------------------------------------
+Download and install the most recent Python v2.7 version of PyWin32 from the
+PyWin32 website. Pick the newest build (with the highest number) and then
+download the appropriate version for your version of Python.
+
+.. _cygwin64install:
+
+Installing Cygwin64
+-------------------
+Download and install the most recent Cygwin64 setup program from the
+`Cygwin website <http://cygwin.com>`_. Now for the most painful part of this guide:
+Picking what Cygwin components you want to install. Unfortunately, there's no
+install all feature in the Cygwin installation program (perhaps because the total
+size is rather large), so you have to use the GUI setup program and manually pick
+these items:
+
+ | Devel/DejaGnu.
+ | Interpreter/tcl.
+ | Tcl/expect.
+ | -- Workaround for test-suite Bug 13121
+ | git
+ | git-completion
+ | git-svn
+ | gnupg
+ | -- optional sshd server (ssh-host-config)
+ | openssh.
+ | cygrunsrv.
+
+Generally, LLVM have several Linux systems with ssh servers. The packages for an
+optional Cygwin sshd server are included so all build machines can be accessed via ssh.
+None of the other installed packages include an sshd server.
+
+**Notice:** If you do not plan to run the test suite, or sshd server, you don't
+need Cygwin. You can build LLVM + Clang with only Subversion, MingwNN, and CMake.
+
+Installing Buildbot (Zorg)
+--------------------------
+If you want to setup your Windows build machine as Buildslave in the LLVM
+Buildbot Infrastructure (Zorg project), complete the Buildslave instructions at
+`Running Buildbot On Windows <http://trac.buildbot.net/wiki/RunningBuildbotOnWindows>`_.
+Several of the required install steps have been done, and it should not
+take long to complete.
+
+The remaining steps to add your Buildslave to the infrastructure project are in the
+:doc:`HowToAddABuilder` document.
+
+**Notice:** Before connecting your Windows Buildslave to a master, complete the
+post-installation steps to ensure that your Windows Builder configuration successfully
+builds the compiler, and any other builds you intend to use with the infrastructure.
+
+Post-Installation Setup
+=======================
+
+After you've installed all of the above, you need a small batch file to set up
+your environment variables. For unix style builds, you only need to set
+up the ``PATH`` environment variable. MSVC and Intel compilers require setup
+scripts that are invoked from existing environment variables. An example for MSVC10
+is included. So, go ahead and create a file named ``setenv.bat`` in the outermost
+directory of your LLVM build tree (``%LLVMDIR%``), and copy and paste the
+``CMD.EXE`` code snippet below into it:
+
+.. code-block:: bat
+
+ @echo off
+ rem This script sets up the paths BEFORE the Windows system tools
+ rem so as to avoid accidentally invoking Windows tools (i.e. sort).
+
+ rem Set up Cygwin as the fifth item in the search path
+ rem path c:\Cygwin\usr\local\bin;c:\Cygwin\usr\bin;c:\Cygwin\bin;%path%
+ if exist c:\cygwin64 path c:\cygwin64\usr\local\bin;c:\cygwin64\usr\bin;c:\cygwin64\bin;%path%
+ if exist c:\cygwin path c:\cygwin\usr\local\bin;c:\cygwin\usr\bin;c:\cygwin\bin;%path%
+
+ rem Set up TortoiseSVN as fifth in search path for SVN.
+ if exist "c:\Program Files\TortoiseSVN\bin" path c:\Program Files\TortoiseSVN\bin;%path%
+
+ rem Set up Mingw32 as the third item in the search path
+ rem Comment out the next line if you're doing 64-bit builds.
+ if exist c:\Mingw32 path c:\Mingw32\bin;c:\Mingw32\msys\1.0\bin;%path%
+
+ rem Set up GnuWin32 as the second item in the search path
+ rem GnuWin32 must come before MingW32 to get the right Make.
+ if exist c:\GnuWin32 path c:\GnuWin32\bin;%path%
+
+ rem Set up Mingw64 as the first item in the search path
+ rem Comment out the next line if you're doing 32-bit builds.
+ if exist c:\Mingw64 path c:\Mingw64\bin;c:\Mingw64\msys\1.0\bin;%path%
+
+ rem Set up MSVC 10 first in the search path to avoid accidentally
+ rem invoking Unix tools during the build (i.e. link).
+ if exist "%VS100COMNTOOLS%\vsvars32.bat" "%VS100COMNTOOLS%\vsvars32.bat"
+
+Building the Project
+====================
+Now the time has come to actually get the sources and then build them.
+
+
+Getting the LLVM and Clang Sources
+----------------------------------
+Open a console window, and change to the directory ``%LLVMDIR%`` where you
+will be building LLVM:
+
+.. code-block:: bat
+
+ cd /d %LLVMDIR%
+ setenv.bat
+
+Now you can get the sources by using this ``CMD.EXE`` batch file - copy and
+paste it to a batch file with the name ``llvmcheckout.bat`` in the
+``%LLVMDIR%`` directory:
+
+.. code-block:: bat
+
+ @echo off
+ rem Pulls (checks out) all LLVM and Clang sources from llvm.org
+
+ rem Checkout/Update LLVM
+ if exist llvm-trunk goto updateLLVM
+ pushd .
+ svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm-trunk
+ if errorlevel 1 goto Error
+ popd
+ goto endLLVM
+ :updateLLVM
+ pushd llvm-trunk
+ svn update -q
+ if errorlevel 1 goto Error
+ popd
+ :endLLVM
+
+ rem Checkout/Update Clang
+ if exist llvm-trunk\tools\clang goto updateCLANG
+ pushd llvm-trunk\tools
+ svn co http://llvm.org/svn/llvm-project/cfe/trunk clang
+ if errorlevel 1 goto Error
+ popd
+ goto endCLANG
+ :updateCLANG
+ pushd llvm-trunk\tools\clang
+ svn update -q
+ if errorlevel 1 goto Error
+ popd
+ :endCLANG
+
+ rem Checkout/Update Compiler-RT
+ if exist llvm-trunk\projects\compiler-rt goto updateCOMPRT
+ pushd llvm-trunk\projects
+ svn co http://llvm.org/svn/llvm-project/compiler-rt/trunk compiler-rt
+ if errorlevel 1 goto Error
+ popd
+ goto endCOMPRT
+ :updateCOMPRT
+ pushd llvm-trunk\projects\compiler-rt
+ svn update -q
+ if errorlevel 1 goto Error
+ popd
+ :endCOMPRT
+
+ rem Checkout/Update test-suite
+ rem Bug 13121 - ClamAV test case contains a virus that blocks SVN retrieve (WONTFIX)
+ rem WORKAROUND: Use LLVM GIT mirror instead of SVN.
+ if exist llvm-trunk\projects\test-suite goto updateSUITE
+ pushd llvm-trunk\projects
+ git clone http://llvm.org/git/test-suite test-suite
+ if errorlevel 1 goto Error
+ popd
+ goto endSUITE
+ :updateSUITE
+ pushd llvm-trunk\projects\test-suite
+ git pull
+ if errorlevel 1 goto Error
+ popd
+ :endSUITE
+
+ goto End
+ :Error
+ echo Error at:
+ cd
+ popd
+ echo Error: LLVM pull failed
+ goto End
+
+ :End
+
+Now run the ``llvmcheckout.bat`` command:
+
+.. code-block:: bat
+
+ llvmcheckout
+
+The whole thing takes about five minutes so you can safely go get a cup of
+coffee or what you normally do when you have to wait for your machine.
+
+If everything went okay and there were no error messages, you've succesfully
+got all of the relevant sources from the LLVM Subversion repository.
+
+You'll find the newly retrieved source files in ``%LLVMDIR%\llvm-trunk``.
+
+
+Making the Makefiles
+--------------------
+Now you just need to configure the build system so that you can start the
+actual build of LLVM and Clang. You can optionally specify one or more
+CMake configuration parameters:
+
+.. code-block:: bash
+
+ CMAKE_BUILD_TYPE=Release (enables release build mode without debug info)
+ CMAKE_BUILD_TYPE=Debug (enables debug mode with lots of debug info)
+ CMAKE_INSTALL_PREFIX=path (selects where to install LLVM and Clang)
+
+You specify these on the command-line when you invoke ``cmake.exe``:
+
+.. code-block:: bat
+
+ cd /d %LLVMDIR%
+ md llvm-build
+ cd llvm-build
+ cmake CMAKE_BUILD_TYPE=Release CMAKE_INSTALL_PREFIX=..\install -G Ninja ..\llvm-trunk
+
+There's a complete list of CMake variables in the `Building LLVM with CMake
+<http://llvm.org/docs/CMake.html>`_ document.
+
+After you've run CMake the first time, you can use ``cmake-gui`` to configure
+your build. Alternatively, you can simply fire up ``cmake-gui`` instead of
+``cmake`` as the fourth command in the code snippet shown above.
+
+**Notice:** ``cmake-gui`` is not part of the special Ninja version of CMake.
+
+
+Building the Sources
+--------------------
+You are now ready to build LLVM and Clang using Ninja:
+
+.. code-block:: bat
+
+ %%LLVMDIR%%\ninja
+
+This step takes about 15 minutes on a 1st generation Intel Core i7 with four
+cores.
+
+Using the Built Product
+-----------------------
+Once you have succesfully built LLVM and Clang, you can find the binaries in
+the ``bin`` directory. Preferredly, you can do a ``ninja install`` to make
+the binaries, libraries, and headers, and have them installed in the location
+you specified as part of the ``CMAKE_INSTALL_PREFIX`` variable above.
+
+
+Updating the Source Tree
+------------------------
+Occasionally (read: as often as possible), you want to synchronize with the
+main trunk version of LLVM and Clang so that you discover incompatibilities
+and errors as early as possible. To do that, you can use this script (copy
+and paste it to ``%LLVMDIR%\llvmupdate.bat``):
+
+.. code-block:: bat
+
+ @echo off
+ rem Updates all LLVM and Clang sources from llvm.org
+
+ pushd llvm-trunk
+ svn update
+ if errorlevel 1 goto Error
+
+ cd tools\clang
+ if errorlevel 1 goto Error
+ svn update
+ if errorlevel 1 goto Error
+
+ cd ..\..\projects\compiler-rt
+ if errorlevel 1 goto Error
+ svn update
+ if errorlevel 1 goto Error
+
+ cd ..\test-suite
+ git pull
+ if errorlevel 1 goto Error
+
+ popd
+ if errorlevel 1 goto Error
+
+ goto End
+
+ :Error
+ echo Error: LLVM update failed
+ goto End
+
+ :End
+
+You should run that script as often as your work permits you to. Please keep
+in mind that running the script will require you to issue a new ``ninja``
+command to update the build tree.
+
+**Notice:** ``ninja update`` does not work, and you need the above script.
+
+Appendix A: Building LLVM and Clang using Clang
+===============================================
+Let's assume that you want to build LLVM and Clang using Clang itself. How do
+you do that? The process is very simple, but takes a while to discover.
+
+**Notice:** Depending on where you installed CMake, you may need Administrator
+rights. The easiest is to use ``Windows Explorer`` to create the files with
+as it automatically elevates your rights when needed.
+
+Here goes:
+
+1. Locate your installation of CMake: ``which cmake.exe``
+2. Change to the directory where CMake is installed:
+ ``cd /d <directory-part-of-output-of-which-command>``
+3. Change to the Platform directory: ``cd ..\share\cmake-2.8\Modules\Platform``
+4. Create the file ``Windows-Clang-C.cmake`` and copy and paste this into it:
+
+.. code-block:: cmake
+
+ if(MINGW)
+ include(Platform/Windows-GNU)
+ __windows_compiler_gnu(C)
+ else()
+ # Chain to generic Windows configuration
+ include(Platform/Windows)
+ endif()
+
+5. Create the file ``Windows-Clang-CXX.cmake`` and copy/paste this into it:
+
+.. code-block:: cmake
+
+ if(MINGW)
+ include(Platform/Windows-GNU)
+ __windows_compiler_gnu(C)
+ else()
+ # Chain to generic Windows configuration
+ include(Platform/Windows)
+ endif()
+
+6. Now type each of these commands in succession:
+
+.. code-block:: bat
+
+ set PATH=%LLVMDIR%\install;%PATH%
+ set CC=clang
+ set CXX=clang++
+ cd /d %LLVMDIR%
+ md llvm-clang
+ cd llvm-clang
+ CMake -G Ninja ..\llvm-trunk
+
+7. Launch the Ninja utility to build using Clang++:
+
+.. code-block:: bat
+
+ ninja
+
+The modifications to CMake fixes the problem that CMake currently does not
+support Clang on Windows that well. It is expected to be fixed in the next
+version of CMake.
+
+You should now be building LLVM and Clang using Clang itself.
+
+
+Appendix B: Debugging MinGW Programs on Windows
+===============================================
+You basically have four choices for debugging MinGW programs on Windows:
+
+1. GDB - The GNU Debugger. This is mostly intended for emergencies.
+2. DDD - The Data Display Debugger. A GUI debugger that requires you to
+ install Cygwin.
+3. ADG - `The Affinic Debugger GUI <http://www.affinic.com/?page_id=109>`_.
+ This needs Cygwin64 ``gdb``, which supports 64-bit debugging.
+4. ``printf()`` or ``cout``. This method is guaranteed to work in most
+ environments but requires efficient debugging techniques by the user.
+
+Once you've built all of LLVM and Clang using debug symbols, the default
+for Subversion sources, you should be able to fire up the debugger you've
+chosen and simply pass the path of the program you want to debug, and any
+arguments to it, as the arguments following the program name:
+
+.. code-block:: bat
+
+ gdb d:\bld\llvm-build\llvm\bin\clang.exe -c test.c
+
+or:
+
+.. code-block:: bat
+
+ ddd d:\bld\llvm-build\llvm\bin\clang.exe -c test.c
+
+or
+
+.. code-block:: bat
+
+ adg d:\bld\llvm-build\llvm\bin\clang.exe -c test.c
+
+Both of the GUI frontends run on top of ``gdb`` for which reason they
+cannot support executable formats not supported by ``gdb``.
+
+
+Appendix C: ``CMakeLists.txt`` Snippet Illustrating Link Order
+==============================================================
+Below you'll find a CMake snippet by Nathan Jeffords that
+illustrates the link order of the LLVM libraries:
+
+.. code-block:: cmake
+
+ ${PLATFORM_LIBS}
+
+ #LLVMX86Disassembler
+ #LLVMX86AsmParser
+ LLVMX86AsmPrinter
+ LLVMX86CodeGen
+ LLVMX86Utils
+ LLVMX86Desc
+ LLVMX86Info
+
+ LLVMSelectionDAG
+
+ LLVMAsmPrinter
+
+ #LLVMJIT
+ #LLVMExecutionEngine
+
+ LLVMCodeGen
+ LLVMScalarOpts
+ LLVMTransformUtils
+
+ #LLVMipa
+ LLVMAnalysis
+ LLVMTarget
+
+ LLVMMCParser
+ LLVMMC
+
+ LLVMCore
+ LLVMSupport
+
+This is important because the GNU linker (``ld``) takes the order of input
+files very seriously and therefore can fail even though all symbols are
+present in the list of libraries given to ``ld``.
+
+You only need to use this list if you run into the problem that a link
+against the LLVM libraries turn up unresolved symbols in the LLVM
+namespace.
+
Index: docs/index.rst
===================================================================
--- docs/index.rst (revision 194431)
+++ docs/index.rst (working copy)
@@ -73,6 +73,7 @@
FAQ
Lexicon
HowToAddABuilder
+ HowToSetupAWindowsBuilder
yaml2obj
HowToSubmitABug
SphinxQuickstartTemplate
@@ -143,6 +144,9 @@
:doc:`HowToAddABuilder`
Instructions for adding new builder to LLVM buildbot master.
+:doc:`HowToSetupAWindowsBuilder`
+ Instructions for configuring a Windows Builder.
+
:doc:`YamlIO`
A reference guide for using LLVM's YAML I/O library.
More information about the llvm-commits
mailing list