[clang-tools-extra] [clang-doc] Wire Markdown parser into MDGenerator for fenced code blocks (PR #201746)

[Paul Kirth via cfe-commits]

================
@@ -0,0 +1,157 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "Markdown.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Allocator.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Support/raw_ostream.h"
+
+#define DEBUG_TYPE "clang-doc-markdown"
+
+namespace clang {
+namespace doc {
+namespace markdown {
+
+static MDNode makeText(llvm::StringRef S) {
+  return {NodeKind::NK_Text, S, {}};
+}
+
+// A line is a table separator if it only contains |, -, :, and spaces,
+// and has at least one -.
+static bool isSepRow(llvm::StringRef Line) {
+  return Line.contains('-') &&
+         Line.find_first_not_of("|-: ") == llvm::StringRef::npos;
+}
+
+static llvm::ArrayRef<MDNode>
+allocateNodes(llvm::SmallVectorImpl<MDNode> &Nodes,
+              llvm::BumpPtrAllocator &Arena) {
+  if (Nodes.empty())
+    return {};
+  MDNode *Allocated = Arena.Allocate<MDNode>(Nodes.size());
+  std::uninitialized_copy(Nodes.begin(), Nodes.end(), Allocated);
+  return llvm::ArrayRef<MDNode>(Allocated, Nodes.size());
+}
+
+llvm::ArrayRef<MDNode> parseMarkdown(llvm::StringRef ParagraphText,
+                                     llvm::BumpPtrAllocator &Arena) {
+  if (ParagraphText.trim().empty()) {
+    LLVM_DEBUG(llvm::dbgs() << "[md] empty input, returning nothing\n");
+    return {};
+  }
+
+  llvm::SmallVector<llvm::StringRef, 16> Lines;
+  ParagraphText.split(Lines, '\n');
+
+  LLVM_DEBUG(llvm::dbgs() << "[md] parsing " << Lines.size() << " line(s)\n");
+
+  llvm::SmallVector<MDNode, 8> Nodes;
+  unsigned I = 0;
+
+  while (I < Lines.size()) {
+    llvm::StringRef Line = Lines[I].trim();
+
+    if (Line.empty()) {
+      ++I;
+      continue;
+    }
+
+    // Fenced code block: ``` or ~~~
+    if (Line.starts_with("```") || Line.starts_with("~~~")) {
----------------
ilovepi wrote:

Are these valid only at the start of a line? For Doxygen, its hard to say IMO: https://www.doxygen.nl/manual/markdown.html

CommonMark seems to have more well documented rules: https://spec.commonmark.org/0.12/#fenced-code-blocks

I'd probably err on the side of CommonMark, given the nicer spec to start and add some TODO/FIXMEs for things that will need different handling for Doxygen.  That's probably an OK general approach, but we should consider these on a case-by case basis.

https://github.com/llvm/llvm-project/pull/201746