[llvm] 210a383 - [LLVM][TableGen] Add jupyter kernel for llvm-tblgen
David Spickett via llvm-commits
llvm-commits at lists.llvm.org
Mon Jan 23 06:16:01 PST 2023
Author: David Spickett
Date: 2023-01-23T14:15:55Z
New Revision: 210a383e18ece7917edb9a7e20cecad4cf0daeb0
URL: https://github.com/llvm/llvm-project/commit/210a383e18ece7917edb9a7e20cecad4cf0daeb0
DIFF: https://github.com/llvm/llvm-project/commit/210a383e18ece7917edb9a7e20cecad4cf0daeb0.diff
LOG: [LLVM][TableGen] Add jupyter kernel for llvm-tblgen
This is based on the MLIR opt kernel:
https://github.com/llvm/llvm-project/tree/main/mlir/utils/jupyter
The inent of this is to enable experimentation and the creation
of interactive tutorials for the basics of tablegen.
Noteable changes from that:
* Removed the codemirror mode settings since those won't exist
for tablegen.
* Added "%args" "magic" to control arguments sent to llvm-tblgen.
(magics are directives, see
https://ipython.readthedocs.io/en/stable/interactive/magics.html)
For example the following:
```
%args --print-detailed-records
class Stuff {}
def water_bottle : Stuff {}
```
Produces:
```
DETAILED RECORDS for file -
-------------------- Global Variables (0) --------------------
-------------------- Classes (1) --------------------
Stuff |<stdin>:1|
Template args: (none)
Superclasses: (none)
Fields: (none)
-------------------- Records (1) --------------------
water_bottle |<stdin>:3|
Superclasses: Stuff
Fields: (none)
```
Reviewed By: jpienaar, awarzynski
Differential Revision: https://reviews.llvm.org/D132378
Added:
llvm/utils/TableGen/jupyter/.gitignore
llvm/utils/TableGen/jupyter/LLVM_TableGen.ipynb
llvm/utils/TableGen/jupyter/LLVM_TableGen.md
llvm/utils/TableGen/jupyter/README.md
llvm/utils/TableGen/jupyter/tablegen_kernel/__init__.py
llvm/utils/TableGen/jupyter/tablegen_kernel/__main__.py
llvm/utils/TableGen/jupyter/tablegen_kernel/assets/kernel.json
llvm/utils/TableGen/jupyter/tablegen_kernel/install.py
llvm/utils/TableGen/jupyter/tablegen_kernel/kernel.py
Modified:
Removed:
################################################################################
diff --git a/llvm/utils/TableGen/jupyter/.gitignore b/llvm/utils/TableGen/jupyter/.gitignore
new file mode 100644
index 0000000000000..3b62e63974120
--- /dev/null
+++ b/llvm/utils/TableGen/jupyter/.gitignore
@@ -0,0 +1,5 @@
+__pycache__
+*.pyc
+build/
+dist/
+MANIFEST
diff --git a/llvm/utils/TableGen/jupyter/LLVM_TableGen.ipynb b/llvm/utils/TableGen/jupyter/LLVM_TableGen.ipynb
new file mode 100644
index 0000000000000..82e5282637868
--- /dev/null
+++ b/llvm/utils/TableGen/jupyter/LLVM_TableGen.ipynb
@@ -0,0 +1,252 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "c45bc113",
+ "metadata": {},
+ "source": [
+ "# LLVM TableGen Kernel"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "483313fc",
+ "metadata": {},
+ "source": [
+ "This notebook is running `llvm-tblgen`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "e238dd42",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "------------- Classes -----------------\n",
+ "class Foo {\n",
+ "}\n",
+ "------------- Defs -----------------\n"
+ ]
+ }
+ ],
+ "source": [
+ "// This is some tablegen\n",
+ "class Foo {}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "27f5aa83",
+ "metadata": {},
+ "source": [
+ "Errors printed to stderr are shown."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "e8b10293",
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "<stdin>:1:1: error: Unexpected token at top level\n",
+ "This is not tablegen.\n",
+ "^\n"
+ ]
+ }
+ ],
+ "source": [
+ "This is not tablegen."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "83907141",
+ "metadata": {},
+ "source": [
+ "Add some classes to get some output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "4ca8a9cb",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "------------- Classes -----------------\n",
+ "class Stuff {\n",
+ "}\n",
+ "------------- Defs -----------------\n",
+ "def thing {\t// Stuff\n",
+ "}\n"
+ ]
+ }
+ ],
+ "source": [
+ "class Stuff {}\n",
+ "def thing : Stuff {}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3f29d1a0",
+ "metadata": {},
+ "source": [
+ "Currently cells are not connected. Meaning that this next cell doesn't have the class from the previous one."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "3a204c70",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "<stdin>:1:19: error: Couldn't find class 'Stuff'\n",
+ "def other_thing : Stuff {}\n",
+ " ^\n"
+ ]
+ }
+ ],
+ "source": [
+ "def other_thing : Stuff {}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f6d4613b",
+ "metadata": {},
+ "source": [
+ "There is a \"magic\" directive `%args` that you can use to send command line arguments to `llvm-tblgen`.\n",
+ "\n",
+ "For example, here we have some code that shows a warning."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "2586893b",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "<stdin>:1:25: warning: unused template argument: Thing:B\n",
+ "class Thing <int A, int B> {\n",
+ " ^\n"
+ ]
+ }
+ ],
+ "source": [
+ "class Thing <int A, int B> {\n",
+ " int num = A;\n",
+ "}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "41be44e7",
+ "metadata": {},
+ "source": [
+ "We can pass an argument to ignore that warning."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "ae078bc4",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "------------- Classes -----------------\n",
+ "class Thing<int Thing:A = ?, int Thing:B = ?> {\n",
+ " int num = Thing:A;\n",
+ "}\n",
+ "------------- Defs -----------------\n"
+ ]
+ }
+ ],
+ "source": [
+ "%args --no-warn-on-unused-template-args\n",
+ "class Thing <int A, int B> {\n",
+ " int num = A;\n",
+ "}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "316b9543",
+ "metadata": {},
+ "source": [
+ "The last `%args` in a cell will be the arguments used."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "9634170c",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "<stdin>:1:25: warning: unused template argument: Thing:B\n",
+ "class Thing <int A, int B> {\n",
+ " ^\n"
+ ]
+ }
+ ],
+ "source": [
+ "%args --no-warn-on-unused-template-args\n",
+ "%args\n",
+ "class Thing <int A, int B> {\n",
+ " int num = A;\n",
+ "}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "29bd004b",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "LLVM TableGen",
+ "language": "tablegen",
+ "name": "tablegen"
+ },
+ "language_info": {
+ "file_extension": ".td",
+ "mimetype": "text/x-tablegen",
+ "name": "tablegen",
+ "pygments_lexer": "text"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/llvm/utils/TableGen/jupyter/LLVM_TableGen.md b/llvm/utils/TableGen/jupyter/LLVM_TableGen.md
new file mode 100644
index 0000000000000..52a6a1de66e96
--- /dev/null
+++ b/llvm/utils/TableGen/jupyter/LLVM_TableGen.md
@@ -0,0 +1,109 @@
+# LLVM TableGen Kernel
+
+This notebook is running `llvm-tblgen`.
+
+
+```tablegen
+// This is some tablegen
+class Foo {}
+```
+
+ ------------- Classes -----------------
+ class Foo {
+ }
+ ------------- Defs -----------------
+
+
+Errors printed to stderr are shown.
+
+
+```tablegen
+This is not tablegen.
+```
+
+ <stdin>:1:1: error: Unexpected token at top level
+ This is not tablegen.
+ ^
+
+
+Add some classes to get some output.
+
+
+```tablegen
+class Stuff {}
+def thing : Stuff {}
+```
+
+ ------------- Classes -----------------
+ class Stuff {
+ }
+ ------------- Defs -----------------
+ def thing { // Stuff
+ }
+
+
+Currently cells are not connected. Meaning that this next cell doesn't have the class from the previous one.
+
+
+```tablegen
+def other_thing : Stuff {}
+```
+
+ <stdin>:1:19: error: Couldn't find class 'Stuff'
+ def other_thing : Stuff {}
+ ^
+
+
+There is a "magic" directive `%args` that you can use to send command line arguments to `llvm-tblgen`.
+
+For example, here we have some code that shows a warning.
+
+
+```tablegen
+class Thing <int A, int B> {
+ int num = A;
+}
+```
+
+ <stdin>:1:25: warning: unused template argument: Thing:B
+ class Thing <int A, int B> {
+ ^
+
+
+We can pass an argument to ignore that warning.
+
+
+```tablegen
+%args --no-warn-on-unused-template-args
+class Thing <int A, int B> {
+ int num = A;
+}
+```
+
+ ------------- Classes -----------------
+ class Thing<int Thing:A = ?, int Thing:B = ?> {
+ int num = Thing:A;
+ }
+ ------------- Defs -----------------
+
+
+The last `%args` in a cell will be the arguments used.
+
+
+```tablegen
+%args --no-warn-on-unused-template-args
+%args
+class Thing <int A, int B> {
+ int num = A;
+}
+```
+
+ <stdin>:1:25: warning: unused template argument: Thing:B
+ class Thing <int A, int B> {
+ ^
+
+
+
+```tablegen
+
+```
diff --git a/llvm/utils/TableGen/jupyter/README.md b/llvm/utils/TableGen/jupyter/README.md
new file mode 100644
index 0000000000000..046aebe408ad2
--- /dev/null
+++ b/llvm/utils/TableGen/jupyter/README.md
@@ -0,0 +1,36 @@
+A Jupyter kernel for TableGen (llvm-tblgen)
+
+To use the kernel, first install it into jupyter:
+
+ python3 -m tablegen_kernel.install
+
+Then put this folder on your PYTHONPATH so jupyter can find it:
+
+```shell
+ export PYTHONPATH=$PYTHONPATH:<path to this dir>
+```
+
+Then run one of:
+
+```shell
+ jupyter notebook
+ # Then in the notebook interface, select 'LLVM TableGen' from the 'New' menu.
+
+ # To run the example notebook in this folder.
+ jupyter notebook LLVM_Tablegen.ipynb
+
+ # To use the kernel from the command line.
+ jupyter console --kernel tablegen
+```
+
+If you just want to see the results of the notebook, you can read
+`LLVM_Tablegen.md` instead.
+
+`llvm-tblgen` is expected to be either in the `PATH` or you can set
+the environment variable `LLVM_TBLGEN_EXECUTABLE` to point to it directly.
+
+To run the kernel's doctests do:
+
+```shell
+ python3 tablegen_kernel/kernel.py
+```
diff --git a/llvm/utils/TableGen/jupyter/tablegen_kernel/__init__.py b/llvm/utils/TableGen/jupyter/tablegen_kernel/__init__.py
new file mode 100644
index 0000000000000..d0d124889f93a
--- /dev/null
+++ b/llvm/utils/TableGen/jupyter/tablegen_kernel/__init__.py
@@ -0,0 +1,6 @@
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+"""An llvm-tblgen kernel for Jupyter"""
+
+from .kernel import __version__
diff --git a/llvm/utils/TableGen/jupyter/tablegen_kernel/__main__.py b/llvm/utils/TableGen/jupyter/tablegen_kernel/__main__.py
new file mode 100644
index 0000000000000..d712365cd7b1f
--- /dev/null
+++ b/llvm/utils/TableGen/jupyter/tablegen_kernel/__main__.py
@@ -0,0 +1,8 @@
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from ipykernel.kernelapp import IPKernelApp
+from .kernel import TableGenKernel
+
+IPKernelApp.launch_instance(kernel_class=TableGenKernel)
diff --git a/llvm/utils/TableGen/jupyter/tablegen_kernel/assets/kernel.json b/llvm/utils/TableGen/jupyter/tablegen_kernel/assets/kernel.json
new file mode 100644
index 0000000000000..656fcbf4b379a
--- /dev/null
+++ b/llvm/utils/TableGen/jupyter/tablegen_kernel/assets/kernel.json
@@ -0,0 +1,14 @@
+{
+ "argv": [
+ "python3", "-m", "tablegen_kernel", "-f", "{connection_file}"
+ ],
+ "display_name": "LLVM TableGen",
+ "language": "tablegen",
+ "language_info": {
+ "name": "tablegen",
+ "codemirror_mode": "tablegen",
+ "mimetype": "text/x-tablegen",
+ "file_extension": ".td",
+ "pygments_lexer": "text"
+ }
+}
diff --git a/llvm/utils/TableGen/jupyter/tablegen_kernel/install.py b/llvm/utils/TableGen/jupyter/tablegen_kernel/install.py
new file mode 100644
index 0000000000000..075c8abfdf664
--- /dev/null
+++ b/llvm/utils/TableGen/jupyter/tablegen_kernel/install.py
@@ -0,0 +1,50 @@
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import os
+import argparse
+from jupyter_client.kernelspec import KernelSpecManager
+
+
+def install_my_kernel_spec(user=True, prefix=None):
+ """Install the kernel spec for user in given prefix."""
+ print("Installing llvm-tblgen IPython kernel spec")
+ pkgroot = os.path.dirname(__file__)
+ KernelSpecManager().install_kernel_spec(
+ os.path.join(pkgroot, "assets"), "tablegen", user=user, prefix=prefix
+ )
+
+
+def _is_root():
+ """Returns whether the current user is root."""
+ try:
+ return os.geteuid() == 0
+ except AttributeError:
+ # Return false wherever unknown.
+ return False
+
+
+def main(argv=None):
+ parser = argparse.ArgumentParser(
+ description="Install KernelSpec for LLVM TableGen Kernel"
+ )
+ prefix_locations = parser.add_mutually_exclusive_group()
+
+ prefix_locations.add_argument(
+ "--user", help="Install in user home directory", action="store_true"
+ )
+ prefix_locations.add_argument(
+ "--prefix", help="Install directory prefix", default=None
+ )
+
+ args = parser.parse_args(argv)
+
+ user = args.user or not _is_root()
+ prefix = args.prefix
+
+ install_my_kernel_spec(user=user, prefix=prefix)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/llvm/utils/TableGen/jupyter/tablegen_kernel/kernel.py b/llvm/utils/TableGen/jupyter/tablegen_kernel/kernel.py
new file mode 100644
index 0000000000000..21d04c8f59294
--- /dev/null
+++ b/llvm/utils/TableGen/jupyter/tablegen_kernel/kernel.py
@@ -0,0 +1,160 @@
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import os
+import shutil
+import subprocess
+import tempfile
+from ipykernel.kernelbase import Kernel
+
+__version__ = "0.0.1"
+
+
+class TableGenKernel(Kernel):
+ """Kernel using llvm-tblgen inside jupyter.
+
+ All input is treated as TableGen unless the first non whitespace character
+ is "%" in which case it is a "magic" line.
+
+ The only magic line supported is "%args". The rest of the line is the
+ arguments passed to llvm-tblgen.
+
+ This is "cell magic" meaning it applies to the whole cell. Therefore
+ it must be the first line, or part of a run of magic lines starting
+ from the first line.
+
+ ```tablgen
+ %args
+ %args --print-records --print-detailed-records
+ class Stuff {
+ string Name;
+ }
+
+ def a_thing : Stuff {}
+ ```
+
+ """
+
+ implementation = "tablegen"
+ implementation_version = __version__
+
+ language_version = __version__
+ language = "tablegen"
+ language_info = {
+ "name": "tablegen",
+ "mimetype": "text/x-tablegen",
+ "file_extension": ".td",
+ "pygments_lexer": "text",
+ }
+
+ def __init__(self, **kwargs):
+ Kernel.__init__(self, **kwargs)
+ self._executable = None
+
+ @property
+ def banner(self):
+ return "llvm-tblgen kernel %s" % __version__
+
+ @property
+ def executable(self):
+ """If this is the first run, search for llvm-tblgen.
+ Otherwise return the cached path to it."""
+ if self._executable is None:
+ path = os.environ.get("LLVM_TBLGEN_EXECUTABLE")
+ if path is not None and os.path.isfile(path) and os.access(path, os.X_OK):
+ self._executable = path
+ else:
+ path = shutil.which("llvm-tblgen")
+ if path is None:
+ raise OSError("llvm-tblgen not found, please see README")
+ self._executable = path
+
+ return self._executable
+
+ def get_magic(self, code):
+ """Given a block of code remove the magic lines from it and return
+ a tuple of the code lines (newline joined) and a list of magic lines
+ with their leading spaces removed.
+
+ Currently we only look for "cell magic" which must be at the start of
+ the cell. Meaning the first line, or a set of lines beginning with %
+ that come before the first non-magic line.
+
+ >>> k.get_magic("")
+ ('', [])
+ >>> k.get_magic("Hello World.\\nHello again.")
+ ('Hello World.\\nHello again.', [])
+ >>> k.get_magic(" %foo a b c")
+ ('', ['%foo a b c'])
+ >>> k.get_magic(" %foo a b c\\n%foo\\nFoo")
+ ('Foo', ['%foo a b c', '%foo'])
+ >>> k.get_magic("Foo\\n%foo\\nFoo")
+ ('Foo\\n%foo\\nFoo', [])
+ >>> k.get_magic("%foo\\n Foo\\n%foo")
+ (' Foo\\n%foo', ['%foo'])
+ >>> k.get_magic("%foo\\n\\n%foo")
+ ('\\n%foo', ['%foo'])
+ >>> k.get_magic("%foo\\n \\n%foo")
+ (' \\n%foo', ['%foo'])
+ """
+ magic_lines = []
+ code_lines = []
+
+ lines = code.splitlines()
+ while lines:
+ line = lines.pop(0)
+ possible_magic = line.lstrip()
+ if possible_magic.startswith("%"):
+ magic_lines.append(possible_magic)
+ else:
+ code_lines = [line, *lines]
+ break
+
+ return "\n".join(code_lines), magic_lines
+
+ def do_execute(
+ self, code, silent, store_history=True, user_expressions=None, allow_stdin=False
+ ):
+ """Execute user code using llvm-tblgen binary."""
+ code, magic = self.get_magic(code)
+
+ extra_args = []
+ for m in magic:
+ if m.startswith("%args"):
+ # Last one in wins
+ extra_args = m.split()[1:]
+
+ with tempfile.TemporaryFile("w+") as f:
+ f.write(code)
+ f.seek(0)
+ got = subprocess.run(
+ [self.executable, *extra_args],
+ stdin=f,
+ stderr=subprocess.PIPE,
+ stdout=subprocess.PIPE,
+ universal_newlines=True,
+ )
+
+ if not silent:
+ if got.stderr:
+ self.send_response(
+ self.iopub_socket, "stream", {"name": "stderr", "text": got.stderr}
+ )
+ else:
+ self.send_response(
+ self.iopub_socket, "stream", {"name": "stdout", "text": got.stdout}
+ )
+
+ return {
+ "status": "ok",
+ "execution_count": self.execution_count,
+ "payload": [],
+ "user_expressions": {},
+ }
+
+
+if __name__ == "__main__":
+ import doctest
+
+ doctest.testmod(extraglobs={"k": TableGenKernel()})
More information about the llvm-commits
mailing list