[libc-commits] [libc] [libc] Updated header_generation.rst (PR #99712)

[via libc-commits]

================
@@ -1,120 +1,190 @@
+.. role:: raw-html(raw)
+    :format: html
+
 Generating Public and Internal headers
 ======================================
 
-.. warning::
-  This page is severely out of date. Much of the information it contains may be
-  incorrect. Please only remove this warning once the page has been updated.
-
-Other libc implementations make use of preprocessor macro tricks to make header
-files platform agnostic. When macros aren't suitable, they rely on build
-system tricks to pick the right set of files to compile and export. While these
-approaches have served them well, parts of their systems have become extremely
-complicated making it hard to modify, extend or maintain. To avoid these
-problems in llvm-libc, we use a header generation mechanism. The mechanism is
-driven by a *header configuration language*.
-
-Header Configuration Language
------------------------------
-
-Header configuration language consists of few special *commands*. The header
-generation mechanism takes an input file, which has an extension of
-``.h.def``, and produces a header file with ``.h`` extension. The header
-configuration language commands are listed in the input ``.h.def`` file. While
-reading a ``.h.def`` file, the header generation tool does two things:
-
-1. Copy the lines not containing commands as is into the output ``.h`` file.
-2. Replace the line on which a command occurs with some other text as directed
-   by the command. The replacement text can span multiple lines.
-
-Command syntax
-~~~~~~~~~~~~~~
-
-A command should be listed on a line by itself, and should not span more than
-one line. The first token to appear on the line is the command name prefixed
-with ``%%``. For example, a line with the ``include_file`` command should start
-with ``%%include_file``. There can be indentation spaces before the ``%%``
-prefix.
-
-Most commands typically take arguments. They are listed as a comma separated
-list of named identifiers within parenthesis, similar to the C function call
-syntax. Before performing the action corresponding to the command, the header
-generator replaces the arguments with concrete values.
-
-Argument Syntax
-~~~~~~~~~~~~~~~
-
-Arguments are named indentifiers but prefixed with ``$`` and enclosed in ``{``
-and ``}``. For example, ``${path_to_constants}``.
-
-Comments
-~~~~~~~~
-
-There can be cases wherein one wants to add comments in the .h.def file but
-does not want them to be copied into the generated header file. Such comments
-can be added by beginning the comment lines with the ``<!>`` prefix. Currently,
-comments have to be on lines of their own. That is, they cannot be suffixes like
-this:
-
-```
-%%include_file(a/b/c) <!> Path to c in b of a.  !!! WRONG SYNTAX
-```
-
-Available Commands
-------------------
-
-Sub-sections below describe the commands currently available. Under each command
-is the description of the arguments to the command, and the action taken by the
-header generation tool when processing a command.
-
-``include_file``
-~~~~~~~~~~~~~~~~
-
-This is a replacement command which should be listed in an input ``.h.def``
-file.
-
-Arguments
-
-  * **path argument** - An argument representing a path to a file. The file
-    should have an extension of ``.h.inc``.
-
-Action
-
-  This command instructs that the line on which the command appears should be
-  replaced by the contents of the file whose path is passed as argument to the
-  command.
-
-``begin``
-~~~~~~~~~
-
-This is not a replacement command. It is an error to list it in the input
-``.h.def`` file. It is normally listed in the files included by the
-``include_file`` command (the ``.h.inc`` files). A common use of this command it
-mark the beginning of what is to be included. This prevents copying items like
-license headers into the generated header file.
-
-Arguments
-
-  None.
-
-Action
-
-  The header generator will only include content starting from the line after the
-  line on which this command is listed.
-
-``public_api``
-~~~~~~~~~~~~~~
-
-This is a replacement command which should be listed in an input ``.h.def``
-file. The header file generator will replace this command with the public API of
-the target platform. See the build system document for more information on the
-relevant build rules. Also, see "Mechanics of public_api" to learn the mechanics
-of how the header generator replaces this command with the public API.
-
-Arguments
-
-  None.
-
-Action
-
-  The header generator will replace this command with the public API to be exposed
-  from the generated header file.
+This is a new implementation of the previous libc header generator. The old
+header generator (libc-hdrgen aka "headergen") was based on tablegen, which
+created an awkward dependency on the rest of LLVM for our build system. By
+creating a new standalone headergen we can eliminate these dependencies for
+easier cross compatibility.
+
+There are 3 main components of the new Headergen. The first component are the
+yaml files that contain all the function header information and are separated by
+header specification and standard. The second component are the classes that are
+created for each component of the function header: macros, enumerations, types,
+function, arguments, and objects. The third component is the python script that
+uses the class representation to deserialize yaml files into its specific
+components and then reserializes the components into the function header. The
+python script also combines the generated header content with header definitions
+and extra macro and type inclusions from the .h.def file.
+
+
+Instructions
+------------
+
+Required Versions:
+
+- Python Version: 3.11.8
+- CMake Version: 3.20.0
+
+1. Make sure to have `LLVM <https://llvm.org/docs/GettingStarted.html>`_ on your
+   system.
+2. Make sure you have created a build directory within your LLVM directory.
+3. When generating the necessary files by your build make sure to include:
+   ``-DLLVM_LIBC_FULL_BUILD=ON`` within the CMake command since new headergen
+   only runs on full-build.
+4. Once the build is complete, enter in the command line within the build
+   directory ``ninja check-newhdrgen`` to ensure that the integration tests are
+   passing.
+5. Then enter in the command line ``ninja libc`` to generate headers. Headers
+   will be in ``build/projects/libc/include`` or ``build/libc/include`` in a runtime
+   build. Sys spec headers will be located in
+   ``build/projects/libc/include/sys``.
+
+
+New Headergen is turned on by default, but if you want to use old headergen,
+you can include this statement when building: ``-DLIBC_USE_NEW_HEADER_GEN=OFF``
+
+To add a function to the yaml files, you can either manually enter it in the
+yaml file corresponding to the header it belongs to or add it through the
+command line.
+
+To add through the command line:
+
+1. Make sure you are in the llvm-project directory.
+
+2. Enter in the command line:
+   :raw-html:`<br />` ``python3 libc/newhdrgen/yaml_to_classes.py
+   libc/newhdrgen/yaml/[yaml_file.yaml] --add_function "<return_type>"
+   <function_name> "<function_arg1, function_arg2>" <standard> <guard>
+   <attribute>``
+
+   Example:
+   :raw-html:`<br />` ``python3 libc/newhdrgen/yaml_to_classes.py
+   libc/newhdrgen/yaml/ctype.yaml --add_function "char" example_funtion "int,
+   void, const void" stdc example_float example_attribute``
+   
+   Keep in mind only the return_type and arguments have quotes around them. If
+   you do not have any guards or attributes you may enter "null" for both.
+
+3. Check the yaml file that the added function is present. You will also get a
+   generated header file with the new addition in the newhdrgen directory to
+   examine.
+
+
+Testing
+-------
+
+New Headergen has an integration test that you may run once you have configured
+your CMake within the build directory. In the command line, enter the following:
+``ninja check-newhdrgen``. The integration test is one test that ensures the
+process of yaml to classes to generate headers works properly. If there are any
+new additions on formatting headers, make sure the test is updated with the
+specific addition.
+
+Integration Test can be found in: ``libc/newhdrgen/tests/test_integration.py``
+
+File to modify if adding something to formatting:
+``libc/newhdrgen/tests/expected_output/test_header.h``
+
+
+Common Errors
+-------------
+1. Missing function specific component
+   
+  Example:
+  :raw-html:`<br />` ``"/llvm-project/libc/newhdrgen/yaml_to_classes.py", line
+  67, in yaml_to_classes function_data["return_type"]``
+
+  If you receive this error or any error pertaining to
+  ``function_data[function_specific_component]`` while building the headers
+  that means the function specific component is missing within the yaml files.
+  Through the call stack, you will be able to find the header file which has the
+  issue. Ensure there is no missing function specific component for that yaml
+  header file.
+
+2. CMake Error: require argument to be specified
+
+  Example:
+  :raw-html:`<br />` ``CMake Error at:
+  /llvm-project/libc/cmake/modules/LLVMLibCHeaderRules.cmake:86 (message):``
+  :raw-html:`<br />` ``'add_gen_hdr2' rule requires GEN_HDR to be specified.``
+  :raw-html:`<br />` ``Call Stack (most recent call first):
+  /llvm-project/libc/include/CMakeLists.txt:22 (add_gen_header2)
+  /llvm-project/libc/include/CMakeLists.txt:62 (add_header_macro)``
----------------
overmighty wrote:

Nit: why not a single non-inline code block?

https://github.com/llvm/llvm-project/pull/99712