[libc-commits] [libc] [libc] Rewrite "Full Host Build" instructions (PR #180439)

[Jeff Bailey via libc-commits]

================
@@ -57,162 +45,181 @@ documentation and verbose cmake logging, which are useful development features.
       -DLLVM_ENABLE_SPHINX=ON -DLIBC_INCLUDE_DOCS=ON \
       -DLIBC_CMAKE_VERBOSE_LOGGING=ON
 
-Build and test
-==============
+After configuring the build, you can build the libc, math library, and run the
+tests with the following command:
 
-After configuring the build with the above ``cmake`` command, one can build test
-libc with the following command:
+.. code-block:: sh
+
+   ninja libc libm check-libc
+
+To run a specific unit test for a function, you can target it directly using its
+full name:
 
 .. code-block:: sh
 
-   $> ninja libc libm check-libc
+   ninja -C build libc.test.src.<HEADER>.<FUNCTION>_test.__unit__
+
+For example, to run the test for ``isalpha`` in ``ctype.h``:
+
+.. code-block:: sh
 
-To build the docs run this command:
+   ninja -C build libc.test.src.ctype.isalpha_test.__unit__
 
+Building Documentation
+======================
+
+If you have Sphinx installed, you can build the libc documentation locally. The
+build configuration above already includes the necessary flags
+(``-DLLVM_ENABLE_SPHINX=ON -DLIBC_INCLUDE_DOCS=ON``).
+
+To generate the HTML documentation:
 
 .. code-block:: sh
 
-   $> ninja docs-libc-html
+   ninja -C build docs-libc-html
+
+The generated documentation will be available in the ``docs/libc/html`` directory
+within your build folder.
+
+Building a Sysroot
+==================
+
+.. warning::
+   The LLVM libc is missing many critical functions needed to build non-trivial applications. If you
+   are not currently working on porting the libc, we recommend sticking with your system libc. However,
+   ignoring warnings like this are how most of us got into this business. So: Speak friend and enter.
+
+This document describes how to set up a sysroot and a compiler that uses it from
+scratch. These are not full cross-compilation instructions. We make a few
+assumptions:
+
+ * The host and target are the same architecture and OS. For example, building a Linux x86-64 libc on a Linux x86-64 host.
+ * The host has a working and recent clang toolchain. Clang 21 has been tested.
+ * Your container is using Debian Testing or a derived distribution. Other distributions likely work but the package names and paths may differ.
+
+Step 1: Preparation
+-------------------
 
-To run a specific test, use the following:
+First, set up the environment variables for your sysroot path and the major
+version of your host Clang.
 
 .. code-block:: sh
 
-   $> ninja libc.test.src.<HEADER>.<FUNCTION>_test.__unit__
-   $> ninja libc.test.src.ctype.isalpha_test.__unit__ # EXAMPLE
-
-Configure the complete toolchain build
-======================================
-
-For a complete toolchain we recommend creating a *sysroot* (see the documentation
-of the ``--sysroot`` option here:
-`<https://gcc.gnu.org/onlinedocs/gcc/Directory-Options.html>`_) which includes
-not only the components of LLVM's libc, but also a full LLVM only toolchain
-consisting of the `clang <https://clang.llvm.org/>`_ compiler, the
-`lld <https://lld.llvm.org/>`_ linker and the
-`compiler-rt <https://compiler-rt.llvm.org/>`_ runtime libraries. LLVM-libc is
-not quite complete enough to allow using and linking a C++ application against
-a C++ standard library (like libc++). Hence, we do not include
-`libc++ <https://libcxx.llvm.org/>`_ in the sysroot.
-
-.. note:: When the libc is complete enough, we should be able to include
-   `libc++ <https://libcxx.llvm.org/>`_, libcxx-abi and libunwind in the
-   LLVM only toolchain and use them to build and link C++ applications.
-
-Below is the cmake command for a bootstrapping build of LLVM. This will build
-clang and lld with the current system's toolchain, then build compiler-rt and
-LLVM-libc with that freshly built clang. This ensures that LLVM-libc can take
-advantage of the latest clang features and optimizations.
-
-This build also uses Ninja as cmake's generator, and sets lld and compiler-rt as
-the default for the fresh clang. Those settings are recommended, but the build
-should still work without them. The compiler-rt options are required for
-building `Scudo <https://llvm.org/docs/ScudoHardenedAllocator.html>`_ as the
-allocator for LLVM-libc.
+   SYSROOT=$(readlink -f ~/sysroot)
+
+Step 2: Linux Headers
+---------------------
+
+Next, install the Linux kernel headers into your sysroot. For this guide, we'll
+copy the headers from the host system's ``/usr/include`` directory. This
+includes ``linux``, ``asm-generic``, and the architecture-specific ``asm``
+headers.
+
+.. code-block:: sh
+
+   # Create the include directory
+   mkdir -p $SYSROOT/usr/include
+
+   # Copy the header directories
+   cp -R /usr/include/linux $SYSROOT/usr/include/
+   cp -R /usr/include/asm-generic $SYSROOT/usr/include/
+   # Use -L to dereference the asm symlink and copy the actual files
+   cp -R -L /usr/include/asm $SYSROOT/usr/include/
 
 .. note::
-   if your build fails with an error saying the compiler can't find
-   ``<asm/unistd.h>`` or similar then you're probably missing the symlink from
-   ``/usr/include/asm`` to ``/usr/include/<TARGET TRIPLE>/asm``. Installing the
-   ``gcc-multilib`` package creates this symlink, or you can do it manually with
-   this command:
-   ``sudo ln -s /usr/include/<TARGET TRIPLE>/asm /usr/include/asm``
+   For a more production-ready sysroot, you would typically download a specific
+   kernel version and install the headers using ``make headers_install``
+   configured for the target architecture and installation path.
+
+Step 3: Build and Install Runtimes
+----------------------------------
+
+Now, configure the build for LLVM libc and compiler-rt. We're building with
+llvm instead of runtimes because we need to install the
+``clang-resource-headers`` that provide ``stdarg.h``, ``stddef.h`` and others.
 
 .. code-block:: sh
 
-   $> cd llvm-project  # The llvm-project checkout
-   $> mkdir build
-   $> cd build
-   $> SYSROOT=/path/to/sysroot # Remember to set this!
-   $> cmake ../llvm  \
-      -G Ninja  \
+   cmake \
+      -S llvm \
+      -B build-runtimes \
+      -G Ninja \
+      -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF \
+      -DCMAKE_INSTALL_PREFIX=$SYSROOT/usr \
       -DLLVM_ENABLE_PROJECTS="clang;lld"   \
       -DLLVM_ENABLE_RUNTIMES="libc;compiler-rt" \
-      -DCMAKE_BUILD_TYPE=Release  \
-      -DCMAKE_C_COMPILER=clang \
-      -DCMAKE_CXX_COMPILER=clang++ \
       -DLLVM_LIBC_FULL_BUILD=ON \
+      -DLIBC_INCLUDE_DOCS=OFF \
+      -DCMAKE_BUILD_TYPE=Release \
       -DLLVM_LIBC_INCLUDE_SCUDO=ON \
       -DCOMPILER_RT_BUILD_SCUDO_STANDALONE_WITH_LLVM_LIBC=ON \
       -DCOMPILER_RT_BUILD_GWP_ASAN=OFF                       \
       -DCOMPILER_RT_SCUDO_STANDALONE_BUILD_SHARED=OFF        \
-      -DCLANG_DEFAULT_LINKER=lld \
-      -DCLANG_DEFAULT_RTLIB=compiler-rt \
-      -DCMAKE_INSTALL_PREFIX=$SYSROOT
-
-Build and install
-=================
+      -DCOMPILER_RT_BUILD_BUILTINS:BOOL=TRUE \
+      -DCOMPILER_RT_BUILD_CRT:BOOL=TRUE \
+      -DCOMPILER_RT_BUILD_GWP_ASAN:BOOL=FALSE
 
-.. TODO: add this warning to the cmake
-.. warning::
-   Running these install commands without setting a ``$SYSROOT`` will install
-   them into your system include path, which may break your system. If you're
-   just trying to develop libc, then just run ``ninja check-libc`` to build the
-   libc and run the tests. If you've already accidentally installed the headers,
-   you may need to delete them from ``/usr/local/include``.
-
-After configuring the build with the above ``cmake`` command, one can build and
-install the toolchain with
+After configuring, build and install the necessary components:
 
 .. code-block:: sh
 
-   $> ninja install-clang install-builtins install-compiler-rt  \
-      install-core-resource-headers install-libc install-lld
+   ninja -C build-runtimes install-clang-resource-headers install-libc install-compiler-rt install-builtins
 
-or
+Step 4: Configure the Compiler Wrapper
+--------------------------------------
 
-.. code-block:: sh
+To make using the new toolchain easier, you can create a Clang configuration
+file. This allows you to avoid passing long command line arguments every time
+you compile a program.
----------------
kaladron wrote:

Eh, sorry - I missed what you were saying.  I was intentionally not installing clang in the sysroot.  Partly because it wasn't working right (as soon as I add all the sysroot stuff, I start having problems with the library randomly dropping malloc, etc), and also because I'm hoping to just copy most of the clang resource headers also from the running system, and not build clang and lld at all.  I just haven't 100% convinced myself that these would always be the same between linux-llvm and linux-gnu.

https://github.com/llvm/llvm-project/pull/180439