[libc-commits] [libc] [libc] Updated header_generation.rst (PR #99712)
via libc-commits
libc-commits at lists.llvm.org
Fri Jul 19 14:38:57 PDT 2024
https://github.com/RoseZhang03 created https://github.com/llvm/llvm-project/pull/99712
Added new headergen documentation.
>From 9f320fc3cf4e8b41aa6c45a9aee1d94ca8410025 Mon Sep 17 00:00:00 2001
From: Rose Zhang <rosezhang at google.com>
Date: Fri, 19 Jul 2024 21:37:05 +0000
Subject: [PATCH] [libc] Updated header_generation.rst
Added new headergen documentation.
---
libc/docs/dev/header_generation.rst | 232 ++++++++++++++++------------
1 file changed, 136 insertions(+), 96 deletions(-)
diff --git a/libc/docs/dev/header_generation.rst b/libc/docs/dev/header_generation.rst
index ed6127e403ec7..2b0a8b14e60b3 100644
--- a/libc/docs/dev/header_generation.rst
+++ b/libc/docs/dev/header_generation.rst
@@ -1,120 +1,160 @@
+.. role:: raw-html(raw)
+ :format: html
+
Generating Public and Internal headers
======================================
-.. warning::
- This page is severely out of date. Much of the information it contains may be
- incorrect. Please only remove this warning once the page has been updated.
-
-Other libc implementations make use of preprocessor macro tricks to make header
-files platform agnostic. When macros aren't suitable, they rely on build
-system tricks to pick the right set of files to compile and export. While these
-approaches have served them well, parts of their systems have become extremely
-complicated making it hard to modify, extend or maintain. To avoid these
-problems in llvm-libc, we use a header generation mechanism. The mechanism is
-driven by a *header configuration language*.
-
-Header Configuration Language
------------------------------
-
-Header configuration language consists of few special *commands*. The header
-generation mechanism takes an input file, which has an extension of
-``.h.def``, and produces a header file with ``.h`` extension. The header
-configuration language commands are listed in the input ``.h.def`` file. While
-reading a ``.h.def`` file, the header generation tool does two things:
-
-1. Copy the lines not containing commands as is into the output ``.h`` file.
-2. Replace the line on which a command occurs with some other text as directed
- by the command. The replacement text can span multiple lines.
-
-Command syntax
-~~~~~~~~~~~~~~
-
-A command should be listed on a line by itself, and should not span more than
-one line. The first token to appear on the line is the command name prefixed
-with ``%%``. For example, a line with the ``include_file`` command should start
-with ``%%include_file``. There can be indentation spaces before the ``%%``
-prefix.
+This is a new implementation of the previous libc header generator. The old
+header generator (libc-hdrgen aka “headergen”) is based on tablegen, which
+creates an awkward dependency on the rest of LLVM for our build system. By
+creating a new standalone headergen we can eliminate these dependencies for
+easier cross compatibility.
-Most commands typically take arguments. They are listed as a comma separated
-list of named identifiers within parenthesis, similar to the C function call
-syntax. Before performing the action corresponding to the command, the header
-generator replaces the arguments with concrete values.
+There are 3 main components of the new Headergen. The first component are the
+yaml files that contain all the function header information and are separated by
+header specification and standard. The second component are the classes that are
+created for each component of the function header: macros, enumerations, types,
+function, arguments, and objects. The third component is the python script that
+uses the class representation to deserialize yaml files into its specific
+components and then reserializes the components into the function header. The
+python script also combines the generated header content with header definitions
+and extra macro and type inclusions from the .h.def file.
-Argument Syntax
-~~~~~~~~~~~~~~~
-Arguments are named indentifiers but prefixed with ``$`` and enclosed in ``{``
-and ``}``. For example, ``${path_to_constants}``.
-
-Comments
-~~~~~~~~
-
-There can be cases wherein one wants to add comments in the .h.def file but
-does not want them to be copied into the generated header file. Such comments
-can be added by beginning the comment lines with the ``<!>`` prefix. Currently,
-comments have to be on lines of their own. That is, they cannot be suffixes like
-this:
-
-```
-%%include_file(a/b/c) <!> Path to c in b of a. !!! WRONG SYNTAX
-```
-
-Available Commands
-------------------
-
-Sub-sections below describe the commands currently available. Under each command
-is the description of the arguments to the command, and the action taken by the
-header generation tool when processing a command.
+Instructions
+-----------------------------
-``include_file``
-~~~~~~~~~~~~~~~~
+Required Versions:
-This is a replacement command which should be listed in an input ``.h.def``
-file.
+- Python Version: 3.11.8 [subject to be lower]
+- CMake Version: 3.20.0
-Arguments
+1. Make sure to have `LLVM <https://llvm.org/docs/GettingStarted.html>`_ on your
+ system.
+2. Make sure you have created a build directory within your LLVM directory.
+3. When generating the necessary files by your build make sure to include:
+ ``-DLLVM_LIBC_FULL_BUILD=ON`` within the CMake command since new headergen
+ only runs on full-build.
+4. Once the build is complete, enter in the command line within the build
+ directory ``ninja check-newhdrgen`` to ensure that the integration tests are
+ passing.
+5. Then enter in the command line ``ninja libc`` to generate headers. Headers
+ will be in “build/projects/libc/include” or “build/libc/include” in a runtime
+ build. Sys spec headers will be located in
+ ``build/projects/libc/include/sys``.
- * **path argument** - An argument representing a path to a file. The file
- should have an extension of ``.h.inc``.
-Action
+New Headergen is turned on by default, but if you wanted to use old headergen,
+you can include this statement when building: ``-DLIBC_USE_NEW_HEADER_GEN=OFF``
- This command instructs that the line on which the command appears should be
- replaced by the contents of the file whose path is passed as argument to the
- command.
+To add a function to the yaml files, you can either manually enter it in the
+yaml file depending on the header spec or enter it through the command line.
-``begin``
-~~~~~~~~~
+To enter through the command line:
-This is not a replacement command. It is an error to list it in the input
-``.h.def`` file. It is normally listed in the files included by the
-``include_file`` command (the ``.h.inc`` files). A common use of this command it
-mark the beginning of what is to be included. This prevents copying items like
-license headers into the generated header file.
+1. Make sure you are in the llvm-project directory.
-Arguments
+2. Enter in the command line:
+ ``python3 libc/newhdrgen/yaml_to_classes.py
+ libc/newhdrgen/yaml/[yaml_file.yaml] --add_function “<return_type>”
+ <function_name> “<function_arg1, function_arg2>” <standard> <guard>
+ <attribute>``
- None.
+ Example:
+ ``python3 libc/newhdrgen/yaml_to_classes.py
+ libc/newhdrgen/yaml/ctype.yaml --add_function “char” example_funtion “int,
+ void, const void” stdc example_float example_attribute``
+
+ Keep in mind only the return_type and arguments have quotes around them. If
+ you do not have any guards or attributes you may enter “null” for both.
-Action
+3. Check the yaml file that the added function is present. You will also get a
+ generated header file with the new addition in the newhdrgen directory to
+ examine.
- The header generator will only include content starting from the line after the
- line on which this command is listed.
-``public_api``
-~~~~~~~~~~~~~~
+Testing
+-----------------------------
-This is a replacement command which should be listed in an input ``.h.def``
-file. The header file generator will replace this command with the public API of
-the target platform. See the build system document for more information on the
-relevant build rules. Also, see "Mechanics of public_api" to learn the mechanics
-of how the header generator replaces this command with the public API.
+New Headergen has an integration test that you may run once you have configured
+your CMake within the build directory. In the command line, enter the following:
+“ninja check -newhdrgen”. The integration test is one test that ensures the
+process of yaml to classes to generate headers works properly. If there are any
+new additions on formatting headers, make sure the test is updated with the
+specific addition.
-Arguments
+Integration Test can be found in: ``libc/newhdrgen/tests/test_integration.py``
- None.
+File to modify if adding something to formatting:
+``libc/newhdrgen/tests/expected_output/test_header.h``
-Action
- The header generator will replace this command with the public API to be exposed
- from the generated header file.
+Common Errors
+-----------------------------
+1. ``"/llvm-project/libc/newhdrgen/yaml_to_classes.py", line 67, in yaml_to_classes function_data["return_type"]``
+
+ If you receive this error or any error pertaining to
+ ``function_data[function_specific_component]`` while building the headers that
+ means the function specific component is missing within the yaml files.
+ Through the call stack, you will be able to find the header file which has the
+ issue. Ensure there is no missing function specific component for that yaml
+ header file.
+
+2. ``CMake Error at:
+ /llvm-project/libc/cmake/modules/LLVMLibCHeaderRules.cmake:86 (message):
+ 'add_gen_hdr2' rule requires GEN_HDR to be specified.
+ Call Stack (most recent call first):
+ /llvm-project/libc/include/CMakeLists.txt:22 (add_gen_header2)
+ /llvm-project/libc/include/CMakeLists.txt:62 (add_header_macro)``
+
+ If you receive this error, there is a missing yaml_file, h_def file, or
+ header name within the ``libc/include/CMakeLists.txt``. The last line in the
+ error call stack will point to the header where there is a specific
+ component missing. Ensure the correct style and required files are present:
+
+ | ``[header_name]``
+ | ``[../libc/newhdrgen/yaml/[yaml_file.yaml]``
+ | ``[header_name.h.def]``
+ | ``[header_name.h]``
+ | ``DEPENDS``
+ | ``{Necessary Depend Files}``
+
+3. ``usage: yaml_to_classes.py [-h] [--output_dir OUTPUT_DIR]
+ [--h_def_file H_DEF_FILE] [--add_function RETURN_TYPE NAME ARGUMENTS
+ STANDARDS GUARD ATTRIBUTES][--e ENTRY_POINTS] [--export-decls] yaml_file
+ yaml_to_classes.py: error: argument --add_function: expected 6 arguments``
+
+ In the process of adding a function, you may run into an issue where the
+ command line is requiring more arguments than what you currently have.Ensure that all components of the new function are filled. Even if you do
+ not have a guard or attribute, make sure to put null in those two areas.
+
+4. ``File "/llvm-project/libc/newhdrgen/header.py", line 60, in __str__ for function in self.functions: AttributeError: 'HeaderFile' object has no attribute 'functions'``
+
+ When running ninja libc in the build directory to generate headers you may
+ receive the error above. Essentially this means that in
+ ``libc/newhdrgen/header.py`` there is a missing attribute named functions.
+ Make sure all function components are defined within this file and there are
+ no missing functions to add these components.
+
+5. ``/llvm-project/build/projects/libc/include/sched.h:20:25: error: unknown type name 'size_t'; did you mean 'time_t'?``
+ :raw-html:`<br />` ``20 | int_sched_getcpucount(size_t, const cpu_set_t*) __NOEXCEPT``
+ :raw-html:`<br />` ``/llvm-project/build/projects/libc/include/llvm-libc-types/time_t.h:15:24: note: 'time_t' declared here``
+ :raw-html:`<br />` ``15 | typedef __INT64_TYPE__ time_t;``
+
+ During the header generation process errors like the one above may occur
+ because there are missing types for a specific header file. Check the yaml
+ file corresponding to the header file and make sure all the necessary types
+ that are being used are input into the types as well. Delete the specific
+ header file from the build folder and re-run ninja libc to ensure the types
+ are being recognized.
+
+6. Test Integration Errors: Sometimes the integration test will fail but that
+ still means the process is working unless the comparison between the output
+ and expected_output is not showing. If that is the case make sure in
+ ``libc/newhdrgen/tests/test_integration.py`` there are no missing arguments
+ that run through the script.
+ If the integration tests are failing due to mismatching of lines or small
+ errors in spacing that is nothing to worry about. If this is happening
+ while you are making a new change to the formatting of the headers, then
+ ensure the expected output file ``libc/newhdrgen/tests/expected_output/test_header.h``
+ has the changes you are applying.
More information about the libc-commits
mailing list