[Mlir-commits] [mlir] [mlir][pygments] add mkdocs instructions and `_all_` addition (PR #181978)

[llvmlistbot at llvm.org]

llvmbot wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Perry Gibson (Wheest)

<details>
<summary>Changes</summary>

I've integrating the pygments into a Bazel + Mkdocs pipeline.  However, for this workflow the file requires an `__all__`.  

Downstream, this means I need to set up the tool with a genrule to insert it (or I could use a patch):

```bazel
genrule(
    name = "mlir_lexer_py",
    srcs = ["@<!-- -->llvm-project//mlir:utils/pygments/mlir_lexer.py"],
    outs = ["mlir_lexer.py"],
    # Pygments requires __all__ for lexer modules loaded via the LEXERS mapping
    cmd = "echo '__all__ = [\"MlirLexer\"]' | cat - $< > $@",
)
```

This upstreams the fix, and adds some instructions for users.

---
Full diff: https://github.com/llvm/llvm-project/pull/181978.diff


2 Files Affected:

- (modified) mlir/utils/pygments/README.md (+37-1) 
- (modified) mlir/utils/pygments/mlir_lexer.py (+2) 


``````````diff
diff --git a/mlir/utils/pygments/README.md b/mlir/utils/pygments/README.md
index 838faceb01b0f..09c99b78d8880 100644
--- a/mlir/utils/pygments/README.md
+++ b/mlir/utils/pygments/README.md
@@ -20,6 +20,42 @@ This will produce highlighted output in the terminal. Other output formats are
 available, see Pygments [documentation](https://pygments.org/docs/) for more
 information.
 
+### MkDocs / Python-Markdown
+
+Create a Markdown extension that registers the lexer via Pygments' LEXERS
+mapping:
+
+```python
+# e.g., docs/pygments/mlir.py
+from markdown import Extension
+import pygments.lexers._mapping as _mapping
+
+def _register_mlir_lexer():
+    if "MlirLexer" not in _mapping.LEXERS:
+        _mapping.LEXERS["MlirLexer"] = (
+            "your.module.path.mlir_lexer",  # adjust to your project
+            "MLIR",
+            ("mlir",),
+            ("*.mlir",),
+            ("text/x-mlir",),
+        )
+
+class MlirHighlightExtension(Extension):
+    def extendMarkdown(self, md):
+        _register_mlir_lexer()
+
+def makeExtension(**kwargs):
+    return MlirHighlightExtension(**kwargs)
+```
+
+Add to `mkdocs.yml` (before `pymdownx.highlight`):
+
+```yaml
+markdown_extensions:
+  - your.module.path.mlir
+  - pymdownx.highlight
+```
+
 ### LaTeX Usage
 
 First, make sure your distribution includes the `minted` package and list in
@@ -31,7 +67,7 @@ the preamble.
 
 Place the `mlir_lexer.py` in a place where the `latex` binary can find it,
 typically in the working directory next to the main `.tex` file. Note that you
-will have to invoke `latex` with the `-shell-escape` flag. See the `minted` 
+will have to invoke `latex` with the `-shell-escape` flag. See the `minted`
 package [documentation](https://ctan.org/pkg/minted?lang=en) for more
 information.
 
diff --git a/mlir/utils/pygments/mlir_lexer.py b/mlir/utils/pygments/mlir_lexer.py
index 4cbe0fe236fc4..2c81e207d906b 100644
--- a/mlir/utils/pygments/mlir_lexer.py
+++ b/mlir/utils/pygments/mlir_lexer.py
@@ -6,6 +6,8 @@
 from pygments.token import *
 import re
 
+__all__ = ["MlirLexer"]
+
 
 class MlirLexer(RegexLexer):
     """Pygments lexer for MLIR.

``````````

</details>


https://github.com/llvm/llvm-project/pull/181978