[Mlir-commits] [mlir] 97f1c7c - [docs] Register MLIR lexer for Sphinx docs (#206194)

[llvmlistbot at llvm.org]

Author: Reid Kleckner
Date: 2026-06-30T11:46:53-07:00
New Revision: 97f1c7c1850831ef54241f94ee2f1b4411fc04a6

URL: https://github.com/llvm/llvm-project/commit/97f1c7c1850831ef54241f94ee2f1b4411fc04a6
DIFF: https://github.com/llvm/llvm-project/commit/97f1c7c1850831ef54241f94ee2f1b4411fc04a6.diff

LOG: [docs] Register MLIR lexer for Sphinx docs (#206194)

Fixes building `docs-clang-html` when `cmake -DCLANG_ENABLE_CIR=ON`.

Fragments of generated documentation contain code blocks tagged with the
`mlir` language, and that generates Pygments warnings, which fails the
build.

I chose to register a sphinx extension, which in turn registers the MLIR
pygments dialect from utils/docs/llvm_sphinx. The idea here is that
`mlir` syntax highlighting should work in any llvm-project documentation
anywhere (openmp, or anything else).

I'm a little less confident about the change to the mlir lexer itself,
which seems to be necessary in order to get pygments to run without
error. Here is an example error message of what happens if you don't
match whitespace tokens as Text:

```
Warning, treated as error:
/path/to/index.md:3:Lexing literal_block '   %0 = cir.foo : !cir.int<s, 32>\n' as "mlir" resulted in an error at token: ' '. Retrying in relaxed mode.
```

Authored with LLM assistance

Added: 
    utils/docs/llvm_sphinx/ext/mlir_pygments.py

Modified: 
    mlir/utils/pygments/mlir_lexer.py
    utils/docs/llvm_sphinx/__init__.py

Removed: 
    


################################################################################
diff  --git a/mlir/utils/pygments/mlir_lexer.py b/mlir/utils/pygments/mlir_lexer.py
index 2c81e207d906b..c493d20272cc6 100644
--- a/mlir/utils/pygments/mlir_lexer.py
+++ b/mlir/utils/pygments/mlir_lexer.py
@@ -42,6 +42,7 @@ class VariableList(RegexLexer):
 
     tokens = {
         "root": [
+            (r"\s+", Text),
             # Comments
             (r"//.*?$", Comment.Single),
             # operation name with assignment: %... = op.name

diff  --git a/utils/docs/llvm_sphinx/__init__.py b/utils/docs/llvm_sphinx/__init__.py
index 3b63964e5d3c0..8de380aa0b353 100644
--- a/utils/docs/llvm_sphinx/__init__.py
+++ b/utils/docs/llvm_sphinx/__init__.py
@@ -35,7 +35,7 @@ def common_conf(tags: Tags, markdown=Markdown.ALWAYS) -> Dict[str, Any]:
     # needs_sphinx = '1.0'
     # The encoding of source files.
     # source_encoding = 'utf-8-sig'
-    extensions = []
+    extensions = ["llvm_sphinx.ext.mlir_pygments"]
     source_suffix = {".rst": "restructuredtext"}
     if markdown != Markdown.NEVER:
         # When building man pages, we do not use the markdown pages,

diff  --git a/utils/docs/llvm_sphinx/ext/mlir_pygments.py b/utils/docs/llvm_sphinx/ext/mlir_pygments.py
new file mode 100644
index 0000000000000..6b1fd46c166f3
--- /dev/null
+++ b/utils/docs/llvm_sphinx/ext/mlir_pygments.py
@@ -0,0 +1,40 @@
+# -*- coding: utf-8 -*-
+
+"""Sphinx extension for llvm-project MLIR syntax highlighting."""
+
+import importlib.util
+from pathlib import Path
+import sys
+from typing import Dict
+from llvm_sphinx.help import venv_help
+
+try:
+    from sphinx.application import Sphinx
+except ImportError as err:
+    print(venv_help(err), file=sys.stderr)
+    raise
+
+__version__ = "1.0"
+
+
+def _load_mlir_lexer():
+    lexer_path = (
+        Path(__file__).resolve().parents[4]
+        / "mlir"
+        / "utils"
+        / "pygments"
+        / "mlir_lexer.py"
+    )
+    spec = importlib.util.spec_from_file_location("llvm_sphinx_mlir_lexer", lexer_path)
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module.MlirLexer
+
+
+def setup(app: Sphinx) -> Dict[str, object]:
+    app.add_lexer("mlir", _load_mlir_lexer())
+    return {
+        "version": __version__,
+        "parallel_read_safe": True,
+        "parallel_write_safe": True,
+    }