[clang] [clang][ExtractAPI] Fix assertion on invalid UTF-8 in doc comments (PR #212394)

[via cfe-commits]

llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-clang

Author: Patryk Stefanski (patrykstefanski)

<details>
<summary>Changes</summary>

A doc comment can contain invalid UTF-8. The raw bytes reach serializeDocComment, which assigned them to a llvm::json::Value, tripping its valid-UTF-8 assertion. This is hit by legacy headers that are not UTF-8 encoded, which compile fine but crash -extract-api.

Sanitize with json::isUTF8/fixUTF8 before serializing.

Fixes #<!-- -->212393

---
Full diff: https://github.com/llvm/llvm-project/pull/212394.diff


3 Files Affected:

- (modified) clang/docs/ReleaseNotes.md (+2) 
- (modified) clang/lib/ExtractAPI/Serialization/SymbolGraphSerializer.cpp (+5-1) 
- (added) clang/test/ExtractAPI/invalid_utf8_doc_comment.c (+16) 


``````````diff
diff --git a/clang/docs/ReleaseNotes.md b/clang/docs/ReleaseNotes.md
index ac2688886c1ee..7dfd186abcdc5 100644
--- a/clang/docs/ReleaseNotes.md
+++ b/clang/docs/ReleaseNotes.md
@@ -397,6 +397,8 @@ features cannot lower the translation-unit ABI level;
 
 #### Miscellaneous Clang Crashes Fixed
 - Fixed a crash when instantiating an invalid dependent friend destructor declaration in a class template. (#GH210234)
+- Fixed an assertion failure in `-extract-api` when a documentation comment
+  contains invalid UTF-8. (#GH212393)
 
 ### OpenACC Specific Changes
 
diff --git a/clang/lib/ExtractAPI/Serialization/SymbolGraphSerializer.cpp b/clang/lib/ExtractAPI/Serialization/SymbolGraphSerializer.cpp
index d3df9eb604f27..8edaaec0d1c7b 100644
--- a/clang/lib/ExtractAPI/Serialization/SymbolGraphSerializer.cpp
+++ b/clang/lib/ExtractAPI/Serialization/SymbolGraphSerializer.cpp
@@ -262,7 +262,11 @@ std::optional<Object> serializeDocComment(const DocComment &Comment) {
   Array LinesArray;
   for (const auto &CommentLine : Comment) {
     Object Line;
-    Line["text"] = CommentLine.Text;
+    // Source files are not required to be valid UTF-8. JSON values must be
+    // valid UTF-8, so replace any invalid sequences before serializing.
+    Line["text"] = json::isUTF8(CommentLine.Text)
+                       ? CommentLine.Text
+                       : json::fixUTF8(CommentLine.Text);
     serializeObject(Line, "range",
                     serializeSourceRange(CommentLine.Begin, CommentLine.End));
     LinesArray.emplace_back(std::move(Line));
diff --git a/clang/test/ExtractAPI/invalid_utf8_doc_comment.c b/clang/test/ExtractAPI/invalid_utf8_doc_comment.c
new file mode 100644
index 0000000000000..8532aea704d02
--- /dev/null
+++ b/clang/test/ExtractAPI/invalid_utf8_doc_comment.c
@@ -0,0 +1,16 @@
+// RUN: rm -rf %t
+// RUN: %clang_cc1 -extract-api --pretty-sgf --emit-sgf-symbol-labels-for-testing \
+// RUN:   -triple arm64-apple-macosx -x c-header %s -o %t/output.symbols.json
+// RUN: FileCheck %s --input-file %t/output.symbols.json
+
+// This file is purposefully NOT valid UTF-8. The doc comment below contains a
+// raw 0xD5 byte. Be careful when modifying.
+
+/*! @brief The sender�s storage. */
+int foo(void);
+
+// CHECK:      "docComment": {
+// CHECK-NEXT:   "lines": [
+// CHECK:          "text": "@brief The sender�s storage. "
+// CHECK:        ]
+// CHECK-NEXT: },

``````````

</details>


https://github.com/llvm/llvm-project/pull/212394