[libc-commits] [libc] [libc][hdrgen] Allow to treat hdrgen Python code as a Python module. (PR #128955)
Alexey Samsonov via libc-commits
libc-commits at lists.llvm.org
Wed Feb 26 14:55:14 PST 2025
https://github.com/vonosmas created https://github.com/llvm/llvm-project/pull/128955
Move the hdrgen code under a subdirectory to treat it as a Python module.
This mimics the structure used by llvm/utils/lit and
llvm/utils/mlgo-utils and simplifies integration of hdrgen to the build
system which rely on Python modules. In addition to that, it clarifies
which imports are coming from the hdrgen-specific helpers (e.g. "from
type import ..." becomes "from hdrgen.type import ...".
Leave the entrypoints (top-level main.py and yaml_to_classes.py) as-is:
they can keep being referred by the CMake build system w/o any changes.
>From 905b1108fd8cdbbf3e61c4b33cd5f04cc3ca8a35 Mon Sep 17 00:00:00 2001
From: Alexey Samsonov <samsonov at google.com>
Date: Wed, 26 Feb 2025 13:23:10 -0800
Subject: [PATCH 1/2] [libc][hdrgen] Allow to treat hdrgen Python code as a
Python module.
Move the hdrgen code under a subdirectory to treat it as a Python module.
This mimics the structure used by llvm/utils/lit and
llvm/utils/mlgo-utils and simplifies integration of hdrgen to the build
system which rely on Python modules. In addition to that, it clarifies
which imports are coming from the hdrgen-specific helpers (e.g. "from
type import ..." becomes "from hdrgen.type import ...".
Leave the entrypoints (top-level main.py and yaml_to_classes.py) as-is:
they can keep being referred by the CMake build system w/o any changes.
---
libc/utils/hdrgen/hdrgen/__init__.py | 3 +
libc/utils/hdrgen/{ => hdrgen}/enumeration.py | 0
libc/utils/hdrgen/{ => hdrgen}/function.py | 2 +-
libc/utils/hdrgen/{ => hdrgen}/gpu_headers.py | 2 +-
libc/utils/hdrgen/{ => hdrgen}/header.py | 0
libc/utils/hdrgen/{ => hdrgen}/macro.py | 0
libc/utils/hdrgen/hdrgen/main.py | 134 ++++++++
libc/utils/hdrgen/{ => hdrgen}/object.py | 0
libc/utils/hdrgen/{ => hdrgen}/type.py | 0
.../{ => hdrgen}/yaml_functions_sorted.py | 0
libc/utils/hdrgen/hdrgen/yaml_to_classes.py | 285 ++++++++++++++++++
libc/utils/hdrgen/main.py | 124 +-------
libc/utils/hdrgen/yaml_to_classes.py | 277 +----------------
13 files changed, 429 insertions(+), 398 deletions(-)
create mode 100644 libc/utils/hdrgen/hdrgen/__init__.py
rename libc/utils/hdrgen/{ => hdrgen}/enumeration.py (100%)
rename libc/utils/hdrgen/{ => hdrgen}/function.py (99%)
rename libc/utils/hdrgen/{ => hdrgen}/gpu_headers.py (98%)
rename libc/utils/hdrgen/{ => hdrgen}/header.py (100%)
rename libc/utils/hdrgen/{ => hdrgen}/macro.py (100%)
create mode 100755 libc/utils/hdrgen/hdrgen/main.py
rename libc/utils/hdrgen/{ => hdrgen}/object.py (100%)
rename libc/utils/hdrgen/{ => hdrgen}/type.py (100%)
rename libc/utils/hdrgen/{ => hdrgen}/yaml_functions_sorted.py (100%)
create mode 100644 libc/utils/hdrgen/hdrgen/yaml_to_classes.py
mode change 100755 => 100644 libc/utils/hdrgen/main.py
diff --git a/libc/utils/hdrgen/hdrgen/__init__.py b/libc/utils/hdrgen/hdrgen/__init__.py
new file mode 100644
index 0000000000000..44390e67aacf4
--- /dev/null
+++ b/libc/utils/hdrgen/hdrgen/__init__.py
@@ -0,0 +1,3 @@
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
diff --git a/libc/utils/hdrgen/enumeration.py b/libc/utils/hdrgen/hdrgen/enumeration.py
similarity index 100%
rename from libc/utils/hdrgen/enumeration.py
rename to libc/utils/hdrgen/hdrgen/enumeration.py
diff --git a/libc/utils/hdrgen/function.py b/libc/utils/hdrgen/hdrgen/function.py
similarity index 99%
rename from libc/utils/hdrgen/function.py
rename to libc/utils/hdrgen/hdrgen/function.py
index 28af05f78d897..f039996584e31 100644
--- a/libc/utils/hdrgen/function.py
+++ b/libc/utils/hdrgen/hdrgen/function.py
@@ -8,7 +8,7 @@
import re
from functools import total_ordering
-from type import Type
+from hdrgen.type import Type
# These are the keywords that appear in C type syntax but are not part of the
diff --git a/libc/utils/hdrgen/gpu_headers.py b/libc/utils/hdrgen/hdrgen/gpu_headers.py
similarity index 98%
rename from libc/utils/hdrgen/gpu_headers.py
rename to libc/utils/hdrgen/hdrgen/gpu_headers.py
index 505adfa8fee8a..290bfe0649135 100644
--- a/libc/utils/hdrgen/gpu_headers.py
+++ b/libc/utils/hdrgen/hdrgen/gpu_headers.py
@@ -6,7 +6,7 @@
#
# ==-------------------------------------------------------------------------==#
-from header import HeaderFile
+from hdrgen.header import HeaderFile
class GpuHeaderFile(HeaderFile):
diff --git a/libc/utils/hdrgen/header.py b/libc/utils/hdrgen/hdrgen/header.py
similarity index 100%
rename from libc/utils/hdrgen/header.py
rename to libc/utils/hdrgen/hdrgen/header.py
diff --git a/libc/utils/hdrgen/macro.py b/libc/utils/hdrgen/hdrgen/macro.py
similarity index 100%
rename from libc/utils/hdrgen/macro.py
rename to libc/utils/hdrgen/hdrgen/macro.py
diff --git a/libc/utils/hdrgen/hdrgen/main.py b/libc/utils/hdrgen/hdrgen/main.py
new file mode 100755
index 0000000000000..25df41e506a1f
--- /dev/null
+++ b/libc/utils/hdrgen/hdrgen/main.py
@@ -0,0 +1,134 @@
+#!/usr/bin/env python3
+#
+# ===- Generate headers for libc functions ------------------*- python -*--==#
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ==------------------------------------------------------------------------==#
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+from hdrgen.header import HeaderFile
+from hdrgen.yaml_to_classes import load_yaml_file, fill_public_api
+
+
+def main():
+ parser = argparse.ArgumentParser(description="Generate header files from YAML")
+ parser.add_argument(
+ "yaml_file",
+ help="Path to the YAML file containing header specification",
+ metavar="FILE",
+ type=Path,
+ nargs="+",
+ )
+ parser.add_argument(
+ "-o",
+ "--output",
+ help="Path to write generated header file",
+ type=Path,
+ required=True,
+ )
+ parser.add_argument(
+ "--json",
+ help="Write JSON instead of a header, can use multiple YAML files",
+ action="store_true",
+ )
+ parser.add_argument(
+ "--depfile",
+ help="Path to write a depfile",
+ type=Path,
+ )
+ parser.add_argument(
+ "--write-if-changed",
+ help="Write the output file only if its contents have changed",
+ action="store_true",
+ default=False,
+ )
+ parser.add_argument(
+ "-e",
+ "--entry-point",
+ help="Entry point to include; may be given many times",
+ metavar="SYMBOL",
+ action="append",
+ )
+ args = parser.parse_args()
+
+ if not args.json and len(args.yaml_file) != 1:
+ print("Only one YAML file at a time without --json", file=sys.stderr)
+ parser.print_usage(sys.stderr)
+ return 2
+
+ files_read = set()
+
+ def write_depfile():
+ if not args.depfile:
+ return
+ deps = " ".join(str(f) for f in sorted(files_read))
+ args.depfile.parent.mkdir(parents=True, exist_ok=True)
+ with open(args.depfile, "w") as depfile:
+ depfile.write(f"{args.output}: {deps}\n")
+
+ def load_yaml(path):
+ files_read.add(path)
+ return load_yaml_file(path, HeaderFile, args.entry_point)
+
+ def load_header(yaml_file):
+ merge_from_files = dict()
+
+ def merge_from(paths):
+ for path in paths:
+ # Load each file exactly once, in case of redundant merges.
+ if path in merge_from_files:
+ continue
+ header = load_yaml(path)
+ merge_from_files[path] = header
+ merge_from(path.parent / f for f in header.merge_yaml_files)
+
+ # Load the main file first.
+ header = load_yaml(yaml_file)
+
+ # Now load all the merge_yaml_files, and transitive merge_yaml_files.
+ merge_from(yaml_file.parent / f for f in header.merge_yaml_files)
+
+ # Merge in all those files' contents.
+ for merge_from_path, merge_from_header in merge_from_files.items():
+ if merge_from_header.name is not None:
+ print(
+ f"{merge_from_path!s}: Merge file cannot have header field",
+ file=sys.stderr,
+ )
+ return 2
+ header.merge(merge_from_header)
+
+ return header
+
+ if args.json:
+ contents = json.dumps(
+ [load_header(file).json_data() for file in args.yaml_file],
+ indent=2,
+ )
+ else:
+ [yaml_file] = args.yaml_file
+ header = load_header(yaml_file)
+ # The header_template path is relative to the containing YAML file.
+ template = header.template(yaml_file.parent, files_read)
+ contents = fill_public_api(header.public_api(), template)
+
+ write_depfile()
+
+ if (
+ not args.write_if_changed
+ or not args.output.exists()
+ or args.output.read_text() != contents
+ ):
+ args.output.parent.mkdir(parents=True, exist_ok=True)
+ args.output.write_text(contents)
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/libc/utils/hdrgen/object.py b/libc/utils/hdrgen/hdrgen/object.py
similarity index 100%
rename from libc/utils/hdrgen/object.py
rename to libc/utils/hdrgen/hdrgen/object.py
diff --git a/libc/utils/hdrgen/type.py b/libc/utils/hdrgen/hdrgen/type.py
similarity index 100%
rename from libc/utils/hdrgen/type.py
rename to libc/utils/hdrgen/hdrgen/type.py
diff --git a/libc/utils/hdrgen/yaml_functions_sorted.py b/libc/utils/hdrgen/hdrgen/yaml_functions_sorted.py
similarity index 100%
rename from libc/utils/hdrgen/yaml_functions_sorted.py
rename to libc/utils/hdrgen/hdrgen/yaml_functions_sorted.py
diff --git a/libc/utils/hdrgen/hdrgen/yaml_to_classes.py b/libc/utils/hdrgen/hdrgen/yaml_to_classes.py
new file mode 100644
index 0000000000000..ebe7781d449f7
--- /dev/null
+++ b/libc/utils/hdrgen/hdrgen/yaml_to_classes.py
@@ -0,0 +1,285 @@
+#!/usr/bin/env python3
+#
+# ===- Generate headers for libc functions -------------------*- python -*--==#
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ==-------------------------------------------------------------------------==#
+
+import yaml
+import argparse
+from pathlib import Path
+
+from hdrgen.enumeration import Enumeration
+from hdrgen.function import Function
+from hdrgen.gpu_headers import GpuHeaderFile as GpuHeader
+from hdrgen.header import HeaderFile
+from hdrgen.macro import Macro
+from hdrgen.object import Object
+from hdrgen.type import Type
+
+
+def yaml_to_classes(yaml_data, header_class, entry_points=None):
+ """
+ Convert YAML data to header classes.
+
+ Args:
+ yaml_data: The YAML data containing header specifications.
+ header_class: The class to use for creating the header.
+ entry_points: A list of specific function names to include in the header.
+
+ Returns:
+ HeaderFile: An instance of HeaderFile populated with the data.
+ """
+ header_name = yaml_data.get("header")
+ header = header_class(header_name)
+ header.template_file = yaml_data.get("header_template")
+ header.standards = yaml_data.get("standards", [])
+ header.merge_yaml_files = yaml_data.get("merge_yaml_files", [])
+
+ for macro_data in yaml_data.get("macros", []):
+ header.add_macro(
+ Macro(
+ macro_data["macro_name"],
+ macro_data.get("macro_value"),
+ macro_data.get("macro_header"),
+ )
+ )
+
+ types = yaml_data.get("types", [])
+ sorted_types = sorted(types, key=lambda x: x["type_name"])
+ for type_data in sorted_types:
+ header.add_type(Type(type_data["type_name"]))
+
+ for enum_data in yaml_data.get("enums", []):
+ header.add_enumeration(
+ Enumeration(enum_data["name"], enum_data.get("value", None))
+ )
+
+ functions = yaml_data.get("functions", [])
+ if entry_points:
+ entry_points_set = set(entry_points)
+ functions = [f for f in functions if f["name"] in entry_points_set]
+ sorted_functions = sorted(functions, key=lambda x: x["name"])
+ guards = []
+ guarded_function_dict = {}
+ for function_data in sorted_functions:
+ guard = function_data.get("guard", None)
+ if guard is None:
+ arguments = [arg["type"] for arg in function_data["arguments"]]
+ attributes = function_data.get("attributes", None)
+ standards = function_data.get("standards", None)
+ header.add_function(
+ Function(
+ function_data["return_type"],
+ function_data["name"],
+ arguments,
+ standards,
+ guard,
+ attributes,
+ )
+ )
+ else:
+ if guard not in guards:
+ guards.append(guard)
+ guarded_function_dict[guard] = []
+ guarded_function_dict[guard].append(function_data)
+ else:
+ guarded_function_dict[guard].append(function_data)
+ sorted_guards = sorted(guards)
+ for guard in sorted_guards:
+ for function_data in guarded_function_dict[guard]:
+ arguments = [arg["type"] for arg in function_data["arguments"]]
+ attributes = function_data.get("attributes", None)
+ standards = function_data.get("standards", None)
+ header.add_function(
+ Function(
+ function_data["return_type"],
+ function_data["name"],
+ arguments,
+ standards,
+ guard,
+ attributes,
+ )
+ )
+
+ objects = yaml_data.get("objects", [])
+ sorted_objects = sorted(objects, key=lambda x: x["object_name"])
+ for object_data in sorted_objects:
+ header.add_object(
+ Object(object_data["object_name"], object_data["object_type"])
+ )
+
+ return header
+
+
+def load_yaml_file(yaml_file, header_class, entry_points):
+ """
+ Load YAML file and convert it to header classes.
+
+ Args:
+ yaml_file: Path to the YAML file.
+ header_class: The class to use for creating the header (HeaderFile or GpuHeader).
+ entry_points: A list of specific function names to include in the header.
+
+ Returns:
+ HeaderFile: An instance of HeaderFile populated with the data.
+ """
+ with yaml_file.open() as f:
+ yaml_data = yaml.safe_load(f)
+ return yaml_to_classes(yaml_data, header_class, entry_points)
+
+
+def fill_public_api(header_str, h_def_content):
+ """
+ Replace the %%public_api() placeholder in the .h.def content with the generated header content.
+
+ Args:
+ header_str: The generated header string.
+ h_def_content: The content of the .h.def file.
+
+ Returns:
+ The final header content with the public API filled in.
+ """
+ header_str = header_str.strip()
+ return h_def_content.replace("%%public_api()", header_str, 1)
+
+
+def parse_function_details(details):
+ """
+ Parse function details from a list of strings and return a Function object.
+
+ Args:
+ details: A list containing function details
+
+ Returns:
+ Function: An instance of Function initialized with the details.
+ """
+ return_type, name, arguments, standards, guard, attributes = details
+ standards = standards.split(",") if standards != "null" else []
+ arguments = [arg.strip() for arg in arguments.split(",")]
+ attributes = attributes.split(",") if attributes != "null" else []
+
+ return Function(
+ return_type=return_type,
+ name=name,
+ arguments=arguments,
+ standards=standards,
+ guard=guard if guard != "null" else None,
+ attributes=attributes if attributes else [],
+ )
+
+
+def add_function_to_yaml(yaml_file, function_details):
+ """
+ Add a function to the YAML file.
+
+ Args:
+ yaml_file: The path to the YAML file.
+ function_details: A list containing function details (return_type, name, arguments, standards, guard, attributes).
+ """
+ new_function = parse_function_details(function_details)
+
+ with open(yaml_file, "r") as f:
+ yaml_data = yaml.safe_load(f)
+ if "functions" not in yaml_data:
+ yaml_data["functions"] = []
+
+ function_dict = {
+ "name": new_function.name,
+ "standards": new_function.standards,
+ "return_type": new_function.return_type,
+ "arguments": [{"type": arg} for arg in new_function.arguments],
+ }
+
+ if new_function.guard:
+ function_dict["guard"] = new_function.guard
+
+ if new_function.attributes:
+ function_dict["attributes"] = new_function.attributes
+
+ insert_index = 0
+ for i, func in enumerate(yaml_data["functions"]):
+ if func["name"] > new_function.name:
+ insert_index = i
+ break
+ else:
+ insert_index = len(yaml_data["functions"])
+
+ yaml_data["functions"].insert(insert_index, function_dict)
+
+ class IndentYamlListDumper(yaml.Dumper):
+ def increase_indent(self, flow=False, indentless=False):
+ return super(IndentYamlListDumper, self).increase_indent(flow, False)
+
+ with open(yaml_file, "w") as f:
+ yaml.dump(
+ yaml_data,
+ f,
+ Dumper=IndentYamlListDumper,
+ default_flow_style=False,
+ sort_keys=False,
+ )
+
+ print(f"Added function {new_function.name} to {yaml_file}")
+
+
+def main():
+ parser = argparse.ArgumentParser(description="Generate header files from YAML")
+ parser.add_argument(
+ "yaml_file", help="Path to the YAML file containing header specification"
+ )
+ parser.add_argument(
+ "--output_dir",
+ help="Directory to output the generated header file",
+ )
+ parser.add_argument(
+ "--add_function",
+ nargs=6,
+ metavar=(
+ "RETURN_TYPE",
+ "NAME",
+ "ARGUMENTS",
+ "STANDARDS",
+ "GUARD",
+ "ATTRIBUTES",
+ ),
+ help="Add a function to the YAML file",
+ )
+ parser.add_argument(
+ "--entry-point",
+ action="append",
+ help="Entry point to include",
+ dest="entry_points",
+ )
+ parser.add_argument(
+ "--export-decls",
+ action="store_true",
+ help="Flag to use GpuHeader for exporting declarations",
+ )
+ args = parser.parse_args()
+
+ if args.add_function:
+ add_function_to_yaml(args.yaml_file, args.add_function)
+
+ header_class = GpuHeader if args.export_decls else HeaderFile
+ header = load_yaml_file(Path(args.yaml_file), header_class, args.entry_points)
+
+ header_str = str(header)
+
+ if args.output_dir:
+ output_file_path = Path(args.output_dir)
+ if output_file_path.is_dir():
+ output_file_path /= f"{Path(args.yaml_file).stem}.h"
+ else:
+ output_file_path = Path(f"{Path(args.yaml_file).stem}.h")
+
+ if args.export_decls:
+ with open(output_file_path, "w") as f:
+ f.write(header_str)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/libc/utils/hdrgen/main.py b/libc/utils/hdrgen/main.py
old mode 100755
new mode 100644
index d5a1c25e7ce20..3e53661573392
--- a/libc/utils/hdrgen/main.py
+++ b/libc/utils/hdrgen/main.py
@@ -8,127 +8,7 @@
#
# ==------------------------------------------------------------------------==#
-import argparse
-import json
-import sys
-from pathlib import Path
-
-from header import HeaderFile
-from yaml_to_classes import load_yaml_file, fill_public_api
-
-
-def main():
- parser = argparse.ArgumentParser(description="Generate header files from YAML")
- parser.add_argument(
- "yaml_file",
- help="Path to the YAML file containing header specification",
- metavar="FILE",
- type=Path,
- nargs="+",
- )
- parser.add_argument(
- "-o",
- "--output",
- help="Path to write generated header file",
- type=Path,
- required=True,
- )
- parser.add_argument(
- "--json",
- help="Write JSON instead of a header, can use multiple YAML files",
- action="store_true",
- )
- parser.add_argument(
- "--depfile",
- help="Path to write a depfile",
- type=Path,
- )
- parser.add_argument(
- "--write-if-changed",
- help="Write the output file only if its contents have changed",
- action="store_true",
- default=False,
- )
- parser.add_argument(
- "-e",
- "--entry-point",
- help="Entry point to include; may be given many times",
- metavar="SYMBOL",
- action="append",
- )
- args = parser.parse_args()
-
- if not args.json and len(args.yaml_file) != 1:
- print("Only one YAML file at a time without --json", file=sys.stderr)
- parser.print_usage(sys.stderr)
- return 2
-
- files_read = set()
-
- def write_depfile():
- if not args.depfile:
- return
- deps = " ".join(str(f) for f in sorted(files_read))
- args.depfile.parent.mkdir(parents=True, exist_ok=True)
- with open(args.depfile, "w") as depfile:
- depfile.write(f"{args.output}: {deps}\n")
-
- def load_yaml(path):
- files_read.add(path)
- return load_yaml_file(path, HeaderFile, args.entry_point)
-
- def load_header(yaml_file):
- merge_from_files = dict()
-
- def merge_from(paths):
- for path in paths:
- # Load each file exactly once, in case of redundant merges.
- if path in merge_from_files:
- continue
- header = load_yaml(path)
- merge_from_files[path] = header
- merge_from(path.parent / f for f in header.merge_yaml_files)
-
- # Load the main file first.
- header = load_yaml(yaml_file)
-
- # Now load all the merge_yaml_files, and transitive merge_yaml_files.
- merge_from(yaml_file.parent / f for f in header.merge_yaml_files)
-
- # Merge in all those files' contents.
- for merge_from_path, merge_from_header in merge_from_files.items():
- if merge_from_header.name is not None:
- print(
- f"{merge_from_path!s}: Merge file cannot have header field",
- file=sys.stderr,
- )
- return 2
- header.merge(merge_from_header)
-
- return header
-
- if args.json:
- contents = json.dumps(
- [load_header(file).json_data() for file in args.yaml_file],
- indent=2,
- )
- else:
- [yaml_file] = args.yaml_file
- header = load_header(yaml_file)
- # The header_template path is relative to the containing YAML file.
- template = header.template(yaml_file.parent, files_read)
- contents = fill_public_api(header.public_api(), template)
-
- write_depfile()
-
- if (
- not args.write_if_changed
- or not args.output.exists()
- or args.output.read_text() != contents
- ):
- args.output.parent.mkdir(parents=True, exist_ok=True)
- args.output.write_text(contents)
-
+from hdrgen.main import main
if __name__ == "__main__":
- sys.exit(main())
+ main()
diff --git a/libc/utils/hdrgen/yaml_to_classes.py b/libc/utils/hdrgen/yaml_to_classes.py
index 14e1f0f32cbbf..531b443925931 100644
--- a/libc/utils/hdrgen/yaml_to_classes.py
+++ b/libc/utils/hdrgen/yaml_to_classes.py
@@ -1,285 +1,14 @@
#!/usr/bin/env python3
#
-# ===- Generate headers for libc functions -------------------*- python -*--==#
+# ===- Generate headers for libc functions ------------------*- python -*--==#
#
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#
-# ==-------------------------------------------------------------------------==#
-
-import yaml
-import argparse
-from pathlib import Path
-
-from enumeration import Enumeration
-from function import Function
-from gpu_headers import GpuHeaderFile as GpuHeader
-from header import HeaderFile
-from macro import Macro
-from object import Object
-from type import Type
-
-
-def yaml_to_classes(yaml_data, header_class, entry_points=None):
- """
- Convert YAML data to header classes.
-
- Args:
- yaml_data: The YAML data containing header specifications.
- header_class: The class to use for creating the header.
- entry_points: A list of specific function names to include in the header.
-
- Returns:
- HeaderFile: An instance of HeaderFile populated with the data.
- """
- header_name = yaml_data.get("header")
- header = header_class(header_name)
- header.template_file = yaml_data.get("header_template")
- header.standards = yaml_data.get("standards", [])
- header.merge_yaml_files = yaml_data.get("merge_yaml_files", [])
-
- for macro_data in yaml_data.get("macros", []):
- header.add_macro(
- Macro(
- macro_data["macro_name"],
- macro_data.get("macro_value"),
- macro_data.get("macro_header"),
- )
- )
-
- types = yaml_data.get("types", [])
- sorted_types = sorted(types, key=lambda x: x["type_name"])
- for type_data in sorted_types:
- header.add_type(Type(type_data["type_name"]))
-
- for enum_data in yaml_data.get("enums", []):
- header.add_enumeration(
- Enumeration(enum_data["name"], enum_data.get("value", None))
- )
-
- functions = yaml_data.get("functions", [])
- if entry_points:
- entry_points_set = set(entry_points)
- functions = [f for f in functions if f["name"] in entry_points_set]
- sorted_functions = sorted(functions, key=lambda x: x["name"])
- guards = []
- guarded_function_dict = {}
- for function_data in sorted_functions:
- guard = function_data.get("guard", None)
- if guard is None:
- arguments = [arg["type"] for arg in function_data["arguments"]]
- attributes = function_data.get("attributes", None)
- standards = function_data.get("standards", None)
- header.add_function(
- Function(
- function_data["return_type"],
- function_data["name"],
- arguments,
- standards,
- guard,
- attributes,
- )
- )
- else:
- if guard not in guards:
- guards.append(guard)
- guarded_function_dict[guard] = []
- guarded_function_dict[guard].append(function_data)
- else:
- guarded_function_dict[guard].append(function_data)
- sorted_guards = sorted(guards)
- for guard in sorted_guards:
- for function_data in guarded_function_dict[guard]:
- arguments = [arg["type"] for arg in function_data["arguments"]]
- attributes = function_data.get("attributes", None)
- standards = function_data.get("standards", None)
- header.add_function(
- Function(
- function_data["return_type"],
- function_data["name"],
- arguments,
- standards,
- guard,
- attributes,
- )
- )
-
- objects = yaml_data.get("objects", [])
- sorted_objects = sorted(objects, key=lambda x: x["object_name"])
- for object_data in sorted_objects:
- header.add_object(
- Object(object_data["object_name"], object_data["object_type"])
- )
-
- return header
-
-
-def load_yaml_file(yaml_file, header_class, entry_points):
- """
- Load YAML file and convert it to header classes.
-
- Args:
- yaml_file: Path to the YAML file.
- header_class: The class to use for creating the header (HeaderFile or GpuHeader).
- entry_points: A list of specific function names to include in the header.
-
- Returns:
- HeaderFile: An instance of HeaderFile populated with the data.
- """
- with yaml_file.open() as f:
- yaml_data = yaml.safe_load(f)
- return yaml_to_classes(yaml_data, header_class, entry_points)
-
-
-def fill_public_api(header_str, h_def_content):
- """
- Replace the %%public_api() placeholder in the .h.def content with the generated header content.
-
- Args:
- header_str: The generated header string.
- h_def_content: The content of the .h.def file.
-
- Returns:
- The final header content with the public API filled in.
- """
- header_str = header_str.strip()
- return h_def_content.replace("%%public_api()", header_str, 1)
-
-
-def parse_function_details(details):
- """
- Parse function details from a list of strings and return a Function object.
-
- Args:
- details: A list containing function details
-
- Returns:
- Function: An instance of Function initialized with the details.
- """
- return_type, name, arguments, standards, guard, attributes = details
- standards = standards.split(",") if standards != "null" else []
- arguments = [arg.strip() for arg in arguments.split(",")]
- attributes = attributes.split(",") if attributes != "null" else []
-
- return Function(
- return_type=return_type,
- name=name,
- arguments=arguments,
- standards=standards,
- guard=guard if guard != "null" else None,
- attributes=attributes if attributes else [],
- )
-
-
-def add_function_to_yaml(yaml_file, function_details):
- """
- Add a function to the YAML file.
-
- Args:
- yaml_file: The path to the YAML file.
- function_details: A list containing function details (return_type, name, arguments, standards, guard, attributes).
- """
- new_function = parse_function_details(function_details)
-
- with open(yaml_file, "r") as f:
- yaml_data = yaml.safe_load(f)
- if "functions" not in yaml_data:
- yaml_data["functions"] = []
-
- function_dict = {
- "name": new_function.name,
- "standards": new_function.standards,
- "return_type": new_function.return_type,
- "arguments": [{"type": arg} for arg in new_function.arguments],
- }
-
- if new_function.guard:
- function_dict["guard"] = new_function.guard
-
- if new_function.attributes:
- function_dict["attributes"] = new_function.attributes
-
- insert_index = 0
- for i, func in enumerate(yaml_data["functions"]):
- if func["name"] > new_function.name:
- insert_index = i
- break
- else:
- insert_index = len(yaml_data["functions"])
-
- yaml_data["functions"].insert(insert_index, function_dict)
-
- class IndentYamlListDumper(yaml.Dumper):
- def increase_indent(self, flow=False, indentless=False):
- return super(IndentYamlListDumper, self).increase_indent(flow, False)
-
- with open(yaml_file, "w") as f:
- yaml.dump(
- yaml_data,
- f,
- Dumper=IndentYamlListDumper,
- default_flow_style=False,
- sort_keys=False,
- )
-
- print(f"Added function {new_function.name} to {yaml_file}")
-
-
-def main():
- parser = argparse.ArgumentParser(description="Generate header files from YAML")
- parser.add_argument(
- "yaml_file", help="Path to the YAML file containing header specification"
- )
- parser.add_argument(
- "--output_dir",
- help="Directory to output the generated header file",
- )
- parser.add_argument(
- "--add_function",
- nargs=6,
- metavar=(
- "RETURN_TYPE",
- "NAME",
- "ARGUMENTS",
- "STANDARDS",
- "GUARD",
- "ATTRIBUTES",
- ),
- help="Add a function to the YAML file",
- )
- parser.add_argument(
- "--entry-point",
- action="append",
- help="Entry point to include",
- dest="entry_points",
- )
- parser.add_argument(
- "--export-decls",
- action="store_true",
- help="Flag to use GpuHeader for exporting declarations",
- )
- args = parser.parse_args()
-
- if args.add_function:
- add_function_to_yaml(args.yaml_file, args.add_function)
-
- header_class = GpuHeader if args.export_decls else HeaderFile
- header = load_yaml_file(Path(args.yaml_file), header_class, args.entry_points)
-
- header_str = str(header)
-
- if args.output_dir:
- output_file_path = Path(args.output_dir)
- if output_file_path.is_dir():
- output_file_path /= f"{Path(args.yaml_file).stem}.h"
- else:
- output_file_path = Path(f"{Path(args.yaml_file).stem}.h")
-
- if args.export_decls:
- with open(output_file_path, "w") as f:
- f.write(header_str)
+# ==------------------------------------------------------------------------==#
+from hdrgen.yaml_to_classes import main
if __name__ == "__main__":
main()
>From 8c47e5967b29dae4533e3e51778ad4330d57297b Mon Sep 17 00:00:00 2001
From: Alexey Samsonov <samsonov at google.com>
Date: Wed, 26 Feb 2025 14:53:28 -0800
Subject: [PATCH 2/2] Update hdrgen documentation to reflect moved files.
---
libc/docs/dev/header_generation.rst | 13 ++++++-------
1 file changed, 6 insertions(+), 7 deletions(-)
diff --git a/libc/docs/dev/header_generation.rst b/libc/docs/dev/header_generation.rst
index a946106fc7097..2ffb4fffe6e0e 100644
--- a/libc/docs/dev/header_generation.rst
+++ b/libc/docs/dev/header_generation.rst
@@ -63,7 +63,7 @@ To add through the command line:
examine.
If you want to sort the functions alphabetically you can check out
-libc/utils/hdrgen/yaml_functions_sorted.py.
+``libc/utils/hdrgen/hdrgen/yaml_functions_sorted.py``.
Testing
@@ -90,7 +90,7 @@ Common Errors
.. code-block:: none
- "/llvm-project/libc/utils/hdrgen/yaml_to_classes.py", line 67, in yaml_to_classes function_data["return_type"]
+ "/llvm-project/libc/utils/hdrgen/hdrgen/yaml_to_classes.py", line 67, in yaml_to_classes function_data["return_type"]
If you receive this error or any error pertaining to
``function_data[function_specific_component]`` while building the headers
@@ -107,9 +107,9 @@ Common Errors
CMake Error at:
/llvm-project/libc/cmake/modules/LLVMLibCHeaderRules.cmake:86 (message):
- 'add_gen_hdr2' rule requires GEN_HDR to be specified.
+ 'add_gen_hdr' rule requires GEN_HDR to be specified.
Call Stack (most recent call first):
- /llvm-project/libc/include/CMakeLists.txt:22 (add_gen_header2)
+ /llvm-project/libc/include/CMakeLists.txt:22 (add_gen_header)
/llvm-project/libc/include/CMakeLists.txt:62 (add_header_macro)
If you receive this error, there is a missing YAML file, h_def file, or
@@ -119,7 +119,6 @@ Common Errors
| ``[header_name]``
| ``[../libc/include/[yaml_file.yaml]``
- | ``[header_name.h.def]``
| ``[header_name.h]``
| ``DEPENDS``
| ``{Necessary Depend Files}``
@@ -148,13 +147,13 @@ Common Errors
.. code-block:: none
- File "/llvm-project/libc/utils/hdrgen/header.py", line 60, in __str__ for
+ File "/llvm-project/libc/utils/hdrgen/hdrgen/header.py", line 60, in __str__ for
function in self.functions: AttributeError: 'HeaderFile' object has no
attribute 'functions'
When running ``ninja libc`` in the build directory to generate headers you
may receive the error above. Essentially this means that in
- ``libc/utils/hdrgen/header.py`` there is a missing attribute named functions.
+ ``libc/utils/hdrgen/hgrgen/header.py`` there is a missing attribute named functions.
Make sure all function components are defined within this file and there are
no missing functions to add these components.
More information about the libc-commits
mailing list