[Mlir-commits] [llvm] [mlir] [docs] Register MLIR lexer for Sphinx docs (PR #206194)

[llvmlistbot at llvm.org]

llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Reid Kleckner (rnk)

<details>
<summary>Changes</summary>

Fixes building `docs-clang-html` when `cmake -DCLANG_ENABLE_CIR=ON`.

Fragments of generated documentation contain code blocks tagged with the `mlir` language, and that generates Pygments warnings, which fails the build.

I chose to register a sphinx extension, which in turn registers the MLIR pygments dialect from utils/docs/llvm_sphinx. The idea here is that `mlir` syntax highlighting should work in any llvm-project documentation anywhere (openmp, or anything else).

I'm a little less confident about the change to the mlir lexer itself, which seems to be necessary in order to get pygments to run without error. Here is an example error message of what happens if you don't match whitespace tokens as Text:

```
Warning, treated as error:
/path/to/index.md:3:Lexing literal_block '   %0 = cir.foo : !cir.int<s, 32>\n' as "mlir" resulted in an error at token: ' '. Retrying in relaxed mode.
```

---
Full diff: https://github.com/llvm/llvm-project/pull/206194.diff


3 Files Affected:

- (modified) mlir/utils/pygments/mlir_lexer.py (+1) 
- (modified) utils/docs/llvm_sphinx/__init__.py (+1-1) 
- (added) utils/docs/llvm_sphinx/ext/mlir_pygments.py (+40) 


``````````diff
diff --git a/mlir/utils/pygments/mlir_lexer.py b/mlir/utils/pygments/mlir_lexer.py
index 2c81e207d906b..c493d20272cc6 100644
--- a/mlir/utils/pygments/mlir_lexer.py
+++ b/mlir/utils/pygments/mlir_lexer.py
@@ -42,6 +42,7 @@ class VariableList(RegexLexer):
 
     tokens = {
         "root": [
+            (r"\s+", Text),
             # Comments
             (r"//.*?$", Comment.Single),
             # operation name with assignment: %... = op.name
diff --git a/utils/docs/llvm_sphinx/__init__.py b/utils/docs/llvm_sphinx/__init__.py
index 3b63964e5d3c0..8de380aa0b353 100644
--- a/utils/docs/llvm_sphinx/__init__.py
+++ b/utils/docs/llvm_sphinx/__init__.py
@@ -35,7 +35,7 @@ def common_conf(tags: Tags, markdown=Markdown.ALWAYS) -> Dict[str, Any]:
     # needs_sphinx = '1.0'
     # The encoding of source files.
     # source_encoding = 'utf-8-sig'
-    extensions = []
+    extensions = ["llvm_sphinx.ext.mlir_pygments"]
     source_suffix = {".rst": "restructuredtext"}
     if markdown != Markdown.NEVER:
         # When building man pages, we do not use the markdown pages,
diff --git a/utils/docs/llvm_sphinx/ext/mlir_pygments.py b/utils/docs/llvm_sphinx/ext/mlir_pygments.py
new file mode 100644
index 0000000000000..6b1fd46c166f3
--- /dev/null
+++ b/utils/docs/llvm_sphinx/ext/mlir_pygments.py
@@ -0,0 +1,40 @@
+# -*- coding: utf-8 -*-
+
+"""Sphinx extension for llvm-project MLIR syntax highlighting."""
+
+import importlib.util
+from pathlib import Path
+import sys
+from typing import Dict
+from llvm_sphinx.help import venv_help
+
+try:
+    from sphinx.application import Sphinx
+except ImportError as err:
+    print(venv_help(err), file=sys.stderr)
+    raise
+
+__version__ = "1.0"
+
+
+def _load_mlir_lexer():
+    lexer_path = (
+        Path(__file__).resolve().parents[4]
+        / "mlir"
+        / "utils"
+        / "pygments"
+        / "mlir_lexer.py"
+    )
+    spec = importlib.util.spec_from_file_location("llvm_sphinx_mlir_lexer", lexer_path)
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module.MlirLexer
+
+
+def setup(app: Sphinx) -> Dict[str, object]:
+    app.add_lexer("mlir", _load_mlir_lexer())
+    return {
+        "version": __version__,
+        "parallel_read_safe": True,
+        "parallel_write_safe": True,
+    }

``````````

</details>


https://github.com/llvm/llvm-project/pull/206194