[clang] [clang-format] Fix template parsing regression for unspaced user-defi… (PR #210630)

[via cfe-commits]

llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-clang-format

Author: johnnyb2543

<details>
<summary>Changes</summary>

This patch resolves a clang-format regression where unspaced User-Defined Literals (UDLs) were being formatted incorrectly.
Root Cause
The lexer failed to correctly classify the preceding token sequence as a single user-defined literal operator. Since FormatTokenLexer::tryMergeUserDefinedLiteral did not properly merge the operator token and its suffix (e.g., _mag), the internal TokenAnnotator evaluated the subsequent < token as a comparison operator.
Changes
- Updated FormatTokenLexer::tryMergeUserDefinedLiteral to ensure proper token merging for UDLs
- Added unit tests in FormatTest.cpp to verify correct spacing behavior and prevent future regressions.

Fixes #<!-- -->210135 

---
Full diff: https://github.com/llvm/llvm-project/pull/210630.diff


2 Files Affected:

- (modified) clang/lib/Format/FormatTokenLexer.cpp (+32) 
- (modified) clang/unittests/Format/FormatTest.cpp (+4-1) 


``````````diff
diff --git a/clang/lib/Format/FormatTokenLexer.cpp b/clang/lib/Format/FormatTokenLexer.cpp
index 0cf01875af833..12815e8cb3679 100644
--- a/clang/lib/Format/FormatTokenLexer.cpp
+++ b/clang/lib/Format/FormatTokenLexer.cpp
@@ -595,6 +595,38 @@ bool FormatTokenLexer::tryMergeUserDefinedLiteral() {
   if (Tokens.size() < 2)
     return false;
 
+         // --- INTERCEPT STRING/CHARACTER UDLs ALREADY MERGED BY THE RAW LEXER ---
+  // --- INTERCEPT STRING/CHARACTER UDLs ALREADY MERGED BY THE RAW LEXER ---
+  // --- INTERCEPT STRING/CHARACTER UDLs ALREADY MERGED BY THE RAW LEXER ---
+  if (Tokens.back()->isOneOf(tok::string_literal, tok::char_constant, tok::numeric_constant) &&
+      Tokens.end()[-2]->is(tok::kw_operator)) {
+
+    FormatToken *OpToken = Tokens[Tokens.size() - 2];
+    FormatToken *LiteralToken = Tokens.back();
+
+           // Strict guard: Do not merge if this is a member access call (e.g., x.operator""_a())
+    if (Tokens.size() >= 3) {
+      FormatToken *PrevToken = Tokens[Tokens.size() - 3];
+      if (PrevToken->isOneOf(tok::period, tok::arrow) ||
+          PrevToken->TokenText == "." || PrevToken->TokenText == "->") {
+        return false;
+      }
+    }
+
+           // Ensure they are touching in the source text
+    if (!LiteralToken->hasWhitespaceBefore()) {
+      OpToken->TokenText = StringRef(OpToken->TokenText.data(),
+                                     OpToken->TokenText.size() + LiteralToken->TokenText.size());
+      OpToken->ColumnWidth += LiteralToken->ColumnWidth;
+
+      OpToken->Tok.setKind(tok::identifier);
+
+      Tokens.erase(&Tokens.back());
+      return true;
+    }
+  }
+
+         // --- ORIGINAL NUMERIC LITERAL LOGIC KICKS IN HERE ---
   auto *First = Tokens.end() - 2;
   auto &Suffix = First[1];
   if (Suffix->hasWhitespaceBefore() || Suffix->TokenText != "$")
diff --git a/clang/unittests/Format/FormatTest.cpp b/clang/unittests/Format/FormatTest.cpp
index ec9ad612832f5..3ee336e6c46ad 100644
--- a/clang/unittests/Format/FormatTest.cpp
+++ b/clang/unittests/Format/FormatTest.cpp
@@ -16,7 +16,10 @@ namespace test {
 namespace {
 
 class FormatTest : public test::FormatTestBase {};
-
+TEST_F(FormatTest, FormatsUserDefinedLiteralTemplates) {
+    verifyFormat("template <char... Cs> auto f() { return operator\"\"_mag<Cs...>(); }");
+    verifyFormat("template <char... Cs> auto f() { return operator \"\"_mag<Cs...>(); }");
+}
 TEST_F(FormatTest, MessUp) {
   EXPECT_EQ("1 2 3", messUp("1 2 3"));
   EXPECT_EQ("1 2 3", messUp("1\n2\n3"));

``````````

</details>


https://github.com/llvm/llvm-project/pull/210630