[libc-commits] [libc] [libc] Updated header_generation.rst (PR #99712)
via libc-commits
libc-commits at lists.llvm.org
Fri Jul 19 16:59:05 PDT 2024
https://github.com/RoseZhang03 updated https://github.com/llvm/llvm-project/pull/99712
>From 9f320fc3cf4e8b41aa6c45a9aee1d94ca8410025 Mon Sep 17 00:00:00 2001
From: Rose Zhang <rosezhang at google.com>
Date: Fri, 19 Jul 2024 21:37:05 +0000
Subject: [PATCH 1/3] [libc] Updated header_generation.rst
Added new headergen documentation.
---
libc/docs/dev/header_generation.rst | 232 ++++++++++++++++------------
1 file changed, 136 insertions(+), 96 deletions(-)
diff --git a/libc/docs/dev/header_generation.rst b/libc/docs/dev/header_generation.rst
index ed6127e403ec7..2b0a8b14e60b3 100644
--- a/libc/docs/dev/header_generation.rst
+++ b/libc/docs/dev/header_generation.rst
@@ -1,120 +1,160 @@
+.. role:: raw-html(raw)
+ :format: html
+
Generating Public and Internal headers
======================================
-.. warning::
- This page is severely out of date. Much of the information it contains may be
- incorrect. Please only remove this warning once the page has been updated.
-
-Other libc implementations make use of preprocessor macro tricks to make header
-files platform agnostic. When macros aren't suitable, they rely on build
-system tricks to pick the right set of files to compile and export. While these
-approaches have served them well, parts of their systems have become extremely
-complicated making it hard to modify, extend or maintain. To avoid these
-problems in llvm-libc, we use a header generation mechanism. The mechanism is
-driven by a *header configuration language*.
-
-Header Configuration Language
------------------------------
-
-Header configuration language consists of few special *commands*. The header
-generation mechanism takes an input file, which has an extension of
-``.h.def``, and produces a header file with ``.h`` extension. The header
-configuration language commands are listed in the input ``.h.def`` file. While
-reading a ``.h.def`` file, the header generation tool does two things:
-
-1. Copy the lines not containing commands as is into the output ``.h`` file.
-2. Replace the line on which a command occurs with some other text as directed
- by the command. The replacement text can span multiple lines.
-
-Command syntax
-~~~~~~~~~~~~~~
-
-A command should be listed on a line by itself, and should not span more than
-one line. The first token to appear on the line is the command name prefixed
-with ``%%``. For example, a line with the ``include_file`` command should start
-with ``%%include_file``. There can be indentation spaces before the ``%%``
-prefix.
+This is a new implementation of the previous libc header generator. The old
+header generator (libc-hdrgen aka “headergen”) is based on tablegen, which
+creates an awkward dependency on the rest of LLVM for our build system. By
+creating a new standalone headergen we can eliminate these dependencies for
+easier cross compatibility.
-Most commands typically take arguments. They are listed as a comma separated
-list of named identifiers within parenthesis, similar to the C function call
-syntax. Before performing the action corresponding to the command, the header
-generator replaces the arguments with concrete values.
+There are 3 main components of the new Headergen. The first component are the
+yaml files that contain all the function header information and are separated by
+header specification and standard. The second component are the classes that are
+created for each component of the function header: macros, enumerations, types,
+function, arguments, and objects. The third component is the python script that
+uses the class representation to deserialize yaml files into its specific
+components and then reserializes the components into the function header. The
+python script also combines the generated header content with header definitions
+and extra macro and type inclusions from the .h.def file.
-Argument Syntax
-~~~~~~~~~~~~~~~
-Arguments are named indentifiers but prefixed with ``$`` and enclosed in ``{``
-and ``}``. For example, ``${path_to_constants}``.
-
-Comments
-~~~~~~~~
-
-There can be cases wherein one wants to add comments in the .h.def file but
-does not want them to be copied into the generated header file. Such comments
-can be added by beginning the comment lines with the ``<!>`` prefix. Currently,
-comments have to be on lines of their own. That is, they cannot be suffixes like
-this:
-
-```
-%%include_file(a/b/c) <!> Path to c in b of a. !!! WRONG SYNTAX
-```
-
-Available Commands
-------------------
-
-Sub-sections below describe the commands currently available. Under each command
-is the description of the arguments to the command, and the action taken by the
-header generation tool when processing a command.
+Instructions
+-----------------------------
-``include_file``
-~~~~~~~~~~~~~~~~
+Required Versions:
-This is a replacement command which should be listed in an input ``.h.def``
-file.
+- Python Version: 3.11.8 [subject to be lower]
+- CMake Version: 3.20.0
-Arguments
+1. Make sure to have `LLVM <https://llvm.org/docs/GettingStarted.html>`_ on your
+ system.
+2. Make sure you have created a build directory within your LLVM directory.
+3. When generating the necessary files by your build make sure to include:
+ ``-DLLVM_LIBC_FULL_BUILD=ON`` within the CMake command since new headergen
+ only runs on full-build.
+4. Once the build is complete, enter in the command line within the build
+ directory ``ninja check-newhdrgen`` to ensure that the integration tests are
+ passing.
+5. Then enter in the command line ``ninja libc`` to generate headers. Headers
+ will be in “build/projects/libc/include” or “build/libc/include” in a runtime
+ build. Sys spec headers will be located in
+ ``build/projects/libc/include/sys``.
- * **path argument** - An argument representing a path to a file. The file
- should have an extension of ``.h.inc``.
-Action
+New Headergen is turned on by default, but if you wanted to use old headergen,
+you can include this statement when building: ``-DLIBC_USE_NEW_HEADER_GEN=OFF``
- This command instructs that the line on which the command appears should be
- replaced by the contents of the file whose path is passed as argument to the
- command.
+To add a function to the yaml files, you can either manually enter it in the
+yaml file depending on the header spec or enter it through the command line.
-``begin``
-~~~~~~~~~
+To enter through the command line:
-This is not a replacement command. It is an error to list it in the input
-``.h.def`` file. It is normally listed in the files included by the
-``include_file`` command (the ``.h.inc`` files). A common use of this command it
-mark the beginning of what is to be included. This prevents copying items like
-license headers into the generated header file.
+1. Make sure you are in the llvm-project directory.
-Arguments
+2. Enter in the command line:
+ ``python3 libc/newhdrgen/yaml_to_classes.py
+ libc/newhdrgen/yaml/[yaml_file.yaml] --add_function “<return_type>”
+ <function_name> “<function_arg1, function_arg2>” <standard> <guard>
+ <attribute>``
- None.
+ Example:
+ ``python3 libc/newhdrgen/yaml_to_classes.py
+ libc/newhdrgen/yaml/ctype.yaml --add_function “char” example_funtion “int,
+ void, const void” stdc example_float example_attribute``
+
+ Keep in mind only the return_type and arguments have quotes around them. If
+ you do not have any guards or attributes you may enter “null” for both.
-Action
+3. Check the yaml file that the added function is present. You will also get a
+ generated header file with the new addition in the newhdrgen directory to
+ examine.
- The header generator will only include content starting from the line after the
- line on which this command is listed.
-``public_api``
-~~~~~~~~~~~~~~
+Testing
+-----------------------------
-This is a replacement command which should be listed in an input ``.h.def``
-file. The header file generator will replace this command with the public API of
-the target platform. See the build system document for more information on the
-relevant build rules. Also, see "Mechanics of public_api" to learn the mechanics
-of how the header generator replaces this command with the public API.
+New Headergen has an integration test that you may run once you have configured
+your CMake within the build directory. In the command line, enter the following:
+“ninja check -newhdrgen”. The integration test is one test that ensures the
+process of yaml to classes to generate headers works properly. If there are any
+new additions on formatting headers, make sure the test is updated with the
+specific addition.
-Arguments
+Integration Test can be found in: ``libc/newhdrgen/tests/test_integration.py``
- None.
+File to modify if adding something to formatting:
+``libc/newhdrgen/tests/expected_output/test_header.h``
-Action
- The header generator will replace this command with the public API to be exposed
- from the generated header file.
+Common Errors
+-----------------------------
+1. ``"/llvm-project/libc/newhdrgen/yaml_to_classes.py", line 67, in yaml_to_classes function_data["return_type"]``
+
+ If you receive this error or any error pertaining to
+ ``function_data[function_specific_component]`` while building the headers that
+ means the function specific component is missing within the yaml files.
+ Through the call stack, you will be able to find the header file which has the
+ issue. Ensure there is no missing function specific component for that yaml
+ header file.
+
+2. ``CMake Error at:
+ /llvm-project/libc/cmake/modules/LLVMLibCHeaderRules.cmake:86 (message):
+ 'add_gen_hdr2' rule requires GEN_HDR to be specified.
+ Call Stack (most recent call first):
+ /llvm-project/libc/include/CMakeLists.txt:22 (add_gen_header2)
+ /llvm-project/libc/include/CMakeLists.txt:62 (add_header_macro)``
+
+ If you receive this error, there is a missing yaml_file, h_def file, or
+ header name within the ``libc/include/CMakeLists.txt``. The last line in the
+ error call stack will point to the header where there is a specific
+ component missing. Ensure the correct style and required files are present:
+
+ | ``[header_name]``
+ | ``[../libc/newhdrgen/yaml/[yaml_file.yaml]``
+ | ``[header_name.h.def]``
+ | ``[header_name.h]``
+ | ``DEPENDS``
+ | ``{Necessary Depend Files}``
+
+3. ``usage: yaml_to_classes.py [-h] [--output_dir OUTPUT_DIR]
+ [--h_def_file H_DEF_FILE] [--add_function RETURN_TYPE NAME ARGUMENTS
+ STANDARDS GUARD ATTRIBUTES][--e ENTRY_POINTS] [--export-decls] yaml_file
+ yaml_to_classes.py: error: argument --add_function: expected 6 arguments``
+
+ In the process of adding a function, you may run into an issue where the
+ command line is requiring more arguments than what you currently have.Ensure that all components of the new function are filled. Even if you do
+ not have a guard or attribute, make sure to put null in those two areas.
+
+4. ``File "/llvm-project/libc/newhdrgen/header.py", line 60, in __str__ for function in self.functions: AttributeError: 'HeaderFile' object has no attribute 'functions'``
+
+ When running ninja libc in the build directory to generate headers you may
+ receive the error above. Essentially this means that in
+ ``libc/newhdrgen/header.py`` there is a missing attribute named functions.
+ Make sure all function components are defined within this file and there are
+ no missing functions to add these components.
+
+5. ``/llvm-project/build/projects/libc/include/sched.h:20:25: error: unknown type name 'size_t'; did you mean 'time_t'?``
+ :raw-html:`<br />` ``20 | int_sched_getcpucount(size_t, const cpu_set_t*) __NOEXCEPT``
+ :raw-html:`<br />` ``/llvm-project/build/projects/libc/include/llvm-libc-types/time_t.h:15:24: note: 'time_t' declared here``
+ :raw-html:`<br />` ``15 | typedef __INT64_TYPE__ time_t;``
+
+ During the header generation process errors like the one above may occur
+ because there are missing types for a specific header file. Check the yaml
+ file corresponding to the header file and make sure all the necessary types
+ that are being used are input into the types as well. Delete the specific
+ header file from the build folder and re-run ninja libc to ensure the types
+ are being recognized.
+
+6. Test Integration Errors: Sometimes the integration test will fail but that
+ still means the process is working unless the comparison between the output
+ and expected_output is not showing. If that is the case make sure in
+ ``libc/newhdrgen/tests/test_integration.py`` there are no missing arguments
+ that run through the script.
+ If the integration tests are failing due to mismatching of lines or small
+ errors in spacing that is nothing to worry about. If this is happening
+ while you are making a new change to the formatting of the headers, then
+ ensure the expected output file ``libc/newhdrgen/tests/expected_output/test_header.h``
+ has the changes you are applying.
>From 8b4406625c8b61445b6537f7fc667f42cd82df4e Mon Sep 17 00:00:00 2001
From: Rose Zhang <rosezhang at google.com>
Date: Fri, 19 Jul 2024 22:46:17 +0000
Subject: [PATCH 2/3] addressed comments and fixed formatting
---
libc/docs/dev/header_generation.rst | 190 ++++++++++++++++------------
1 file changed, 110 insertions(+), 80 deletions(-)
diff --git a/libc/docs/dev/header_generation.rst b/libc/docs/dev/header_generation.rst
index 2b0a8b14e60b3..fe7e79682b08b 100644
--- a/libc/docs/dev/header_generation.rst
+++ b/libc/docs/dev/header_generation.rst
@@ -5,8 +5,8 @@ Generating Public and Internal headers
======================================
This is a new implementation of the previous libc header generator. The old
-header generator (libc-hdrgen aka “headergen”) is based on tablegen, which
-creates an awkward dependency on the rest of LLVM for our build system. By
+header generator (libc-hdrgen aka "headergen") was based on tablegen, which
+created an awkward dependency on the rest of LLVM for our build system. By
creating a new standalone headergen we can eliminate these dependencies for
easier cross compatibility.
@@ -22,11 +22,11 @@ and extra macro and type inclusions from the .h.def file.
Instructions
------------------------------
+------------
Required Versions:
-- Python Version: 3.11.8 [subject to be lower]
+- Python Version: 3.11.8
- CMake Version: 3.20.0
1. Make sure to have `LLVM <https://llvm.org/docs/GettingStarted.html>`_ on your
@@ -39,34 +39,35 @@ Required Versions:
directory ``ninja check-newhdrgen`` to ensure that the integration tests are
passing.
5. Then enter in the command line ``ninja libc`` to generate headers. Headers
- will be in “build/projects/libc/include” or “build/libc/include” in a runtime
+ will be in ``build/projects/libc/include`` or ``build/libc/include`` in a runtime
build. Sys spec headers will be located in
``build/projects/libc/include/sys``.
-New Headergen is turned on by default, but if you wanted to use old headergen,
+New Headergen is turned on by default, but if you want to use old headergen,
you can include this statement when building: ``-DLIBC_USE_NEW_HEADER_GEN=OFF``
To add a function to the yaml files, you can either manually enter it in the
-yaml file depending on the header spec or enter it through the command line.
+yaml file corresponding to the header it belongs to or add it through the
+command line.
-To enter through the command line:
+To add through the command line:
1. Make sure you are in the llvm-project directory.
2. Enter in the command line:
- ``python3 libc/newhdrgen/yaml_to_classes.py
- libc/newhdrgen/yaml/[yaml_file.yaml] --add_function “<return_type>”
- <function_name> “<function_arg1, function_arg2>” <standard> <guard>
+ :raw-html:`<br />` ``python3 libc/newhdrgen/yaml_to_classes.py
+ libc/newhdrgen/yaml/[yaml_file.yaml] --add_function "<return_type>"
+ <function_name> "<function_arg1, function_arg2>" <standard> <guard>
<attribute>``
Example:
- ``python3 libc/newhdrgen/yaml_to_classes.py
- libc/newhdrgen/yaml/ctype.yaml --add_function “char” example_funtion “int,
- void, const void” stdc example_float example_attribute``
+ :raw-html:`<br />` ``python3 libc/newhdrgen/yaml_to_classes.py
+ libc/newhdrgen/yaml/ctype.yaml --add_function "char" example_funtion "int,
+ void, const void" stdc example_float example_attribute``
Keep in mind only the return_type and arguments have quotes around them. If
- you do not have any guards or attributes you may enter “null” for both.
+ you do not have any guards or attributes you may enter "null" for both.
3. Check the yaml file that the added function is present. You will also get a
generated header file with the new addition in the newhdrgen directory to
@@ -74,11 +75,11 @@ To enter through the command line:
Testing
------------------------------
+-------
New Headergen has an integration test that you may run once you have configured
your CMake within the build directory. In the command line, enter the following:
-“ninja check -newhdrgen”. The integration test is one test that ensures the
+``ninja check-newhdrgen``. The integration test is one test that ensures the
process of yaml to classes to generate headers works properly. If there are any
new additions on formatting headers, make sure the test is updated with the
specific addition.
@@ -90,71 +91,100 @@ File to modify if adding something to formatting:
Common Errors
------------------------------
-1. ``"/llvm-project/libc/newhdrgen/yaml_to_classes.py", line 67, in yaml_to_classes function_data["return_type"]``
-
- If you receive this error or any error pertaining to
- ``function_data[function_specific_component]`` while building the headers that
- means the function specific component is missing within the yaml files.
- Through the call stack, you will be able to find the header file which has the
- issue. Ensure there is no missing function specific component for that yaml
- header file.
-
-2. ``CMake Error at:
- /llvm-project/libc/cmake/modules/LLVMLibCHeaderRules.cmake:86 (message):
- 'add_gen_hdr2' rule requires GEN_HDR to be specified.
- Call Stack (most recent call first):
- /llvm-project/libc/include/CMakeLists.txt:22 (add_gen_header2)
- /llvm-project/libc/include/CMakeLists.txt:62 (add_header_macro)``
-
- If you receive this error, there is a missing yaml_file, h_def file, or
- header name within the ``libc/include/CMakeLists.txt``. The last line in the
- error call stack will point to the header where there is a specific
- component missing. Ensure the correct style and required files are present:
-
- | ``[header_name]``
- | ``[../libc/newhdrgen/yaml/[yaml_file.yaml]``
- | ``[header_name.h.def]``
- | ``[header_name.h]``
- | ``DEPENDS``
- | ``{Necessary Depend Files}``
-
-3. ``usage: yaml_to_classes.py [-h] [--output_dir OUTPUT_DIR]
- [--h_def_file H_DEF_FILE] [--add_function RETURN_TYPE NAME ARGUMENTS
- STANDARDS GUARD ATTRIBUTES][--e ENTRY_POINTS] [--export-decls] yaml_file
- yaml_to_classes.py: error: argument --add_function: expected 6 arguments``
-
- In the process of adding a function, you may run into an issue where the
- command line is requiring more arguments than what you currently have.Ensure that all components of the new function are filled. Even if you do
- not have a guard or attribute, make sure to put null in those two areas.
-
-4. ``File "/llvm-project/libc/newhdrgen/header.py", line 60, in __str__ for function in self.functions: AttributeError: 'HeaderFile' object has no attribute 'functions'``
-
- When running ninja libc in the build directory to generate headers you may
- receive the error above. Essentially this means that in
- ``libc/newhdrgen/header.py`` there is a missing attribute named functions.
- Make sure all function components are defined within this file and there are
- no missing functions to add these components.
-
-5. ``/llvm-project/build/projects/libc/include/sched.h:20:25: error: unknown type name 'size_t'; did you mean 'time_t'?``
- :raw-html:`<br />` ``20 | int_sched_getcpucount(size_t, const cpu_set_t*) __NOEXCEPT``
- :raw-html:`<br />` ``/llvm-project/build/projects/libc/include/llvm-libc-types/time_t.h:15:24: note: 'time_t' declared here``
- :raw-html:`<br />` ``15 | typedef __INT64_TYPE__ time_t;``
-
- During the header generation process errors like the one above may occur
- because there are missing types for a specific header file. Check the yaml
- file corresponding to the header file and make sure all the necessary types
- that are being used are input into the types as well. Delete the specific
- header file from the build folder and re-run ninja libc to ensure the types
- are being recognized.
-
-6. Test Integration Errors: Sometimes the integration test will fail but that
+-------------
+1. Missing function specific component
+
+ Example:
+ :raw-html:`<br />` ``"/llvm-project/libc/newhdrgen/yaml_to_classes.py", line
+ 67, in yaml_to_classes function_data["return_type"]``
+
+ If you receive this error or any error pertaining to
+ ``function_data[function_specific_component]`` while building the headers
+ that means the function specific component is missing within the yaml files.
+ Through the call stack, you will be able to find the header file which has the
+ issue. Ensure there is no missing function specific component for that yaml
+ header file.
+
+2. CMake Error: require argument to be specified
+
+ Example:
+ :raw-html:`<br />` ``CMake Error at:
+ /llvm-project/libc/cmake/modules/LLVMLibCHeaderRules.cmake:86 (message):``
+ :raw-html:`<br />` ``'add_gen_hdr2' rule requires GEN_HDR to be specified.``
+ :raw-html:`<br />` ``Call Stack (most recent call first):
+ /llvm-project/libc/include/CMakeLists.txt:22 (add_gen_header2)
+ /llvm-project/libc/include/CMakeLists.txt:62 (add_header_macro)``
+
+ If you receive this error, there is a missing yaml_file, h_def file, or header
+ name within the ``libc/include/CMakeLists.txt``. The last line in the error
+ call stack will point to the header where there is a specific component
+ missing. Ensure the correct style and required files are present:
+
+ | ``[header_name]``
+ | ``[../libc/newhdrgen/yaml/[yaml_file.yaml]``
+ | ``[header_name.h.def]``
+ | ``[header_name.h]``
+ | ``DEPENDS``
+ | ``{Necessary Depend Files}``
+
+3. Command line: expected arguments
+
+ Example:
+ :raw-html:`<br />` ``usage: yaml_to_classes.py [-h]
+ [--output_dir OUTPUT_DIR] [--h_def_file H_DEF_FILE] [--add_function
+ RETURN_TYPE NAME ARGUMENTS STANDARDS GUARD ATTRIBUTES][--e ENTRY_POINTS]
+ [--export-decls] yaml_file yaml_to_classes.py: error: argument
+ --add_function: expected 6 arguments``
+
+ In the process of adding a function, you may run into an issue where the
+ command line is requiring more arguments than what you currently have. Ensure
+ that all components of the new function are filled. Even if you do not have a
+ guard or attribute, make sure to put null in those two areas.
+
+4. Object has no attribute
+
+ Example:
+ :raw-html:`<br />` ``File "/llvm-project/libc/newhdrgen/header.py", line 60,
+ in __str__ for function in self.functions: AttributeError: 'HeaderFile'
+ object has no attribute 'functions'``
+
+ When running ninja libc in the build directory to generate headers you may
+ receive the error above. Essentially this means that in
+ ``libc/newhdrgen/header.py`` there is a missing attribute named functions.
+ Make sure all function components are defined within this file and there are
+ no missing functions to add these components.
+
+5. Unknown type name
+
+ Example:
+ :raw-html:`<br />`
+ ``/llvm-project/build/projects/libc/include/sched.h:20:25: error: unknown
+ type name 'size_t'; did you mean 'time_t'?``
+ :raw-html:`<br />` ``20 | int_sched_getcpucount(size_t, const cpu_set_t*)
+ __NOEXCEPT``
+ :raw-html:`<br />`
+ ``/llvm-project/build/projects/libc/include/llvm-libc-types/time_t.h:15:24:
+ note: 'time_t' declared here``
+ :raw-html:`<br />` ``15 | typedef __INT64_TYPE__ time_t;``
+
+ During the header generation process errors like the one above may occur
+ because there are missing types for a specific header file. Check the yaml
+ file corresponding to the header file and make sure all the necessary types
+ that are being used are input into the types as well. Delete the specific
+ header file from the build folder and re-run ninja libc to ensure the types
+ are being recognized.
+
+6. Test Integration Errors
+
+ Sometimes the integration test will fail but that
still means the process is working unless the comparison between the output
and expected_output is not showing. If that is the case make sure in
``libc/newhdrgen/tests/test_integration.py`` there are no missing arguments
that run through the script.
+
If the integration tests are failing due to mismatching of lines or small
- errors in spacing that is nothing to worry about. If this is happening
- while you are making a new change to the formatting of the headers, then
- ensure the expected output file ``libc/newhdrgen/tests/expected_output/test_header.h``
- has the changes you are applying.
+ errors in spacing that is nothing to worry about. If this is happening while
+ you are making a new change to the formatting of the headers, then
+ ensure the expected output file
+ ``libc/newhdrgen/tests/expected_output/test_header.h`` has the changes you
+ are applying.
>From 1e046f1cb05c81d85e13d1474a2aef382bab5b60 Mon Sep 17 00:00:00 2001
From: Rose Zhang <rosezhang at google.com>
Date: Fri, 19 Jul 2024 23:58:09 +0000
Subject: [PATCH 3/3] added non-inline code blocks, removed unnecessary
instructions
---
libc/docs/dev/header_generation.rst | 134 ++++++++++++++--------------
1 file changed, 65 insertions(+), 69 deletions(-)
diff --git a/libc/docs/dev/header_generation.rst b/libc/docs/dev/header_generation.rst
index fe7e79682b08b..42cda5664c36f 100644
--- a/libc/docs/dev/header_generation.rst
+++ b/libc/docs/dev/header_generation.rst
@@ -1,23 +1,20 @@
-.. role:: raw-html(raw)
- :format: html
-
Generating Public and Internal headers
======================================
This is a new implementation of the previous libc header generator. The old
-header generator (libc-hdrgen aka "headergen") was based on tablegen, which
+header generator (libc-hdrgen aka "Headergen") was based on Tablegen, which
created an awkward dependency on the rest of LLVM for our build system. By
-creating a new standalone headergen we can eliminate these dependencies for
+creating a new standalone Headergen we can eliminate these dependencies for
easier cross compatibility.
There are 3 main components of the new Headergen. The first component are the
-yaml files that contain all the function header information and are separated by
+YAML files that contain all the function header information and are separated by
header specification and standard. The second component are the classes that are
created for each component of the function header: macros, enumerations, types,
-function, arguments, and objects. The third component is the python script that
-uses the class representation to deserialize yaml files into its specific
+function, arguments, and objects. The third component is the Python script that
+uses the class representation to deserialize YAML files into its specific
components and then reserializes the components into the function header. The
-python script also combines the generated header content with header definitions
+Python script also combines the generated header content with header definitions
and extra macro and type inclusions from the .h.def file.
@@ -27,49 +24,44 @@ Instructions
Required Versions:
- Python Version: 3.11.8
-- CMake Version: 3.20.0
-
-1. Make sure to have `LLVM <https://llvm.org/docs/GettingStarted.html>`_ on your
- system.
-2. Make sure you have created a build directory within your LLVM directory.
-3. When generating the necessary files by your build make sure to include:
- ``-DLLVM_LIBC_FULL_BUILD=ON`` within the CMake command since new headergen
- only runs on full-build.
-4. Once the build is complete, enter in the command line within the build
+
+1. Keep full-build mode on when building, otherwise headers will not be
+ generated.
+2. Once the build is complete, enter in the command line within the build
directory ``ninja check-newhdrgen`` to ensure that the integration tests are
passing.
-5. Then enter in the command line ``ninja libc`` to generate headers. Headers
- will be in ``build/projects/libc/include`` or ``build/libc/include`` in a runtime
- build. Sys spec headers will be located in
+3. Then enter in the command line ``ninja libc`` to generate headers. Headers
+ will be in ``build/projects/libc/include`` or ``build/libc/include`` in a
+ runtime build. Sys spec headers will be located in
``build/projects/libc/include/sys``.
-New Headergen is turned on by default, but if you want to use old headergen,
+New Headergen is turned on by default, but if you want to use old Headergen,
you can include this statement when building: ``-DLIBC_USE_NEW_HEADER_GEN=OFF``
-To add a function to the yaml files, you can either manually enter it in the
-yaml file corresponding to the header it belongs to or add it through the
+To add a function to the YAML files, you can either manually enter it in the
+YAML file corresponding to the header it belongs to or add it through the
command line.
To add through the command line:
1. Make sure you are in the llvm-project directory.
-2. Enter in the command line:
- :raw-html:`<br />` ``python3 libc/newhdrgen/yaml_to_classes.py
- libc/newhdrgen/yaml/[yaml_file.yaml] --add_function "<return_type>"
- <function_name> "<function_arg1, function_arg2>" <standard> <guard>
- <attribute>``
+2. Enter in the command line: ::
+
+ python3 libc/newhdrgen/yaml_to_classes.py
+ libc/newhdrgen/yaml/[yaml_file.yaml] --add_function "<return_type>" <function_name> "<function_arg1, function_arg2>" <standard> <guard> <attribute>
- Example:
- :raw-html:`<br />` ``python3 libc/newhdrgen/yaml_to_classes.py
- libc/newhdrgen/yaml/ctype.yaml --add_function "char" example_funtion "int,
- void, const void" stdc example_float example_attribute``
+ Example: ::
+
+ python3 libc/newhdrgen/yaml_to_classes.py
+ libc/newhdrgen/yaml/ctype.yaml --add_function "char" example_funtion
+ "int, void, const void" stdc example_float example_attribute
Keep in mind only the return_type and arguments have quotes around them. If
you do not have any guards or attributes you may enter "null" for both.
-3. Check the yaml file that the added function is present. You will also get a
+3. Check the YAML file that the added function is present. You will also get a
generated header file with the new addition in the newhdrgen directory to
examine.
@@ -80,7 +72,7 @@ Testing
New Headergen has an integration test that you may run once you have configured
your CMake within the build directory. In the command line, enter the following:
``ninja check-newhdrgen``. The integration test is one test that ensures the
-process of yaml to classes to generate headers works properly. If there are any
+process of YAML to classes to generate headers works properly. If there are any
new additions on formatting headers, make sure the test is updated with the
specific addition.
@@ -94,28 +86,29 @@ Common Errors
-------------
1. Missing function specific component
- Example:
- :raw-html:`<br />` ``"/llvm-project/libc/newhdrgen/yaml_to_classes.py", line
- 67, in yaml_to_classes function_data["return_type"]``
+ Example: ::
+
+ "/llvm-project/libc/newhdrgen/yaml_to_classes.py", line 67, in yaml_to_classes function_data["return_type"]
If you receive this error or any error pertaining to
``function_data[function_specific_component]`` while building the headers
- that means the function specific component is missing within the yaml files.
+ that means the function specific component is missing within the YAML files.
Through the call stack, you will be able to find the header file which has the
- issue. Ensure there is no missing function specific component for that yaml
+ issue. Ensure there is no missing function specific component for that YAML
header file.
2. CMake Error: require argument to be specified
- Example:
- :raw-html:`<br />` ``CMake Error at:
- /llvm-project/libc/cmake/modules/LLVMLibCHeaderRules.cmake:86 (message):``
- :raw-html:`<br />` ``'add_gen_hdr2' rule requires GEN_HDR to be specified.``
- :raw-html:`<br />` ``Call Stack (most recent call first):
- /llvm-project/libc/include/CMakeLists.txt:22 (add_gen_header2)
- /llvm-project/libc/include/CMakeLists.txt:62 (add_header_macro)``
+ Example: ::
- If you receive this error, there is a missing yaml_file, h_def file, or header
+ CMake Error at:
+ /llvm-project/libc/cmake/modules/LLVMLibCHeaderRules.cmake:86 (message):
+ 'add_gen_hdr2' rule requires GEN_HDR to be specified.
+ Call Stack (most recent call first):
+ /llvm-project/libc/include/CMakeLists.txt:22 (add_gen_header2)
+ /llvm-project/libc/include/CMakeLists.txt:62 (add_header_macro)
+
+ If you receive this error, there is a missing YAML file, h_def file, or header
name within the ``libc/include/CMakeLists.txt``. The last line in the error
call stack will point to the header where there is a specific component
missing. Ensure the correct style and required files are present:
@@ -129,12 +122,14 @@ Common Errors
3. Command line: expected arguments
- Example:
- :raw-html:`<br />` ``usage: yaml_to_classes.py [-h]
- [--output_dir OUTPUT_DIR] [--h_def_file H_DEF_FILE] [--add_function
- RETURN_TYPE NAME ARGUMENTS STANDARDS GUARD ATTRIBUTES][--e ENTRY_POINTS]
- [--export-decls] yaml_file yaml_to_classes.py: error: argument
- --add_function: expected 6 arguments``
+ Example: ::
+
+ usage: yaml_to_classes.py [-h] [--output_dir OUTPUT_DIR] [--h_def_file H_DEF_FILE]
+ [--add_function RETURN_TYPE NAME ARGUMENTS STANDARDS GUARD ATTRIBUTES]
+ [--e ENTRY_POINTS] [--export-decls]
+ yaml_file
+ yaml_to_classes.py:
+ error: argument --add_function: expected 6 arguments
In the process of adding a function, you may run into an issue where the
command line is requiring more arguments than what you currently have. Ensure
@@ -143,10 +138,11 @@ Common Errors
4. Object has no attribute
- Example:
- :raw-html:`<br />` ``File "/llvm-project/libc/newhdrgen/header.py", line 60,
- in __str__ for function in self.functions: AttributeError: 'HeaderFile'
- object has no attribute 'functions'``
+ Example: ::
+
+ File "/llvm-project/libc/newhdrgen/header.py", line 60, in __str__ for
+ function in self.functions: AttributeError: 'HeaderFile' object has no
+ attribute 'functions'
When running ninja libc in the build directory to generate headers you may
receive the error above. Essentially this means that in
@@ -156,19 +152,19 @@ Common Errors
5. Unknown type name
- Example:
- :raw-html:`<br />`
- ``/llvm-project/build/projects/libc/include/sched.h:20:25: error: unknown
- type name 'size_t'; did you mean 'time_t'?``
- :raw-html:`<br />` ``20 | int_sched_getcpucount(size_t, const cpu_set_t*)
- __NOEXCEPT``
- :raw-html:`<br />`
- ``/llvm-project/build/projects/libc/include/llvm-libc-types/time_t.h:15:24:
- note: 'time_t' declared here``
- :raw-html:`<br />` ``15 | typedef __INT64_TYPE__ time_t;``
+ Example: ::
+
+ /llvm-project/build/projects/libc/include/sched.h:20:25: error: unknown type
+ name 'size_t'; did you mean 'time_t'?
+ 20 | int_sched_getcpucount(size_t, const cpu_set_t*) __NOEXCEPT
+ | ^
+ /llvm-project/build/projects/libc/include/llvm-libc-types/time_t.h:15:24:
+ note: 'time_t' declared here
+ 15 | typedef __INT64_TYPE__ time_t;
+ | ^
During the header generation process errors like the one above may occur
- because there are missing types for a specific header file. Check the yaml
+ because there are missing types for a specific header file. Check the YAML
file corresponding to the header file and make sure all the necessary types
that are being used are input into the types as well. Delete the specific
header file from the build folder and re-run ninja libc to ensure the types
More information about the libc-commits
mailing list