[PATCH] D134853: [clang-format] Correctly annotate UDLs as OverloadedOperator

[Emilia Dreamer via Phabricator via cfe-commits]

rymiel created this revision.
rymiel added reviewers: HazardyKnusperkeks, owenpan, MyDeveloperDay, curdeius.
Herald added a project: All.
rymiel updated this revision to Diff 463768.
rymiel edited the summary of this revision.
rymiel added a comment.
rymiel updated this revision to Diff 464145.
rymiel updated this revision to Diff 464147.
rymiel published this revision for review.
Herald added a project: clang.
Herald added a subscriber: cfe-commits.

Note issue being fixed


rymiel added a comment.

Notes:

As noted above, the left paren was correctly annotated, but the logic in `rParenEndsCast` goes based off of what is *before* the l paren. This resulted in a paren pair where the left one was OverloadedOperatorLParen and the right one was CastRParen.

This means this bug could really be fixed in two ways, I chose to annotate the UDL with OverloadedOperator, as I felt like that resulted in "more correct tokens".

Edit: While attempting to add token annotator tests for this fix, i noticed there aren't any at all for overloaded operators, and while writing them, I noticed more operators which aren't marked with OverloadedOperator, such as `operator[]` and UDLs which don't start with an underscore. (see below)

Unless any reviewers have any other opinions, I would leave fixing those out of this patch and leave the tests "incomplete" for now?

Also, UDLs that don't start with an underscore aren't considered a single "string_literal" token, instead becoming a string literal `""` and an identifier following it (where as those with an underscore become one token, such as `""_a`). I'm unsure if that's the expected case and if both tokens should just be considered part of the operator


rymiel added a comment.

Also add a few format tests


rymiel added a comment.

Reformat


rymiel added a comment.

While this patch fixes the linked bug, but I encountered more questions that I would like second opinions on (see notes above)


While the opening parenthesis of an user-defined literal operator was
correctly annotated as OverloadedOperatorLParen, the "" and its suffix
wasn't annotated as OverloadedOperator.

Fixes https://github.com/llvm/llvm-project/issues/58035


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D134853

Files:
  clang/lib/Format/TokenAnnotator.cpp
  clang/unittests/Format/FormatTest.cpp
  clang/unittests/Format/TokenAnnotatorTest.cpp


Index: clang/unittests/Format/TokenAnnotatorTest.cpp
===================================================================
--- clang/unittests/Format/TokenAnnotatorTest.cpp
+++ clang/unittests/Format/TokenAnnotatorTest.cpp
@@ -376,6 +376,44 @@
   EXPECT_TOKEN(Tokens[11], tok::amp, TT_PointerOrReference);
 }
 
+TEST_F(TokenAnnotatorTest, UnderstandsOverloadedOperators) {
+  auto Tokens = annotate("x.operator+()");
+  ASSERT_EQ(Tokens.size(), 7u) << Tokens;
+  EXPECT_TOKEN(Tokens[3], tok::plus, TT_OverloadedOperator);
+  EXPECT_TOKEN(Tokens[4], tok::l_paren, TT_OverloadedOperatorLParen);
+  Tokens = annotate("x.operator=()");
+  ASSERT_EQ(Tokens.size(), 7u) << Tokens;
+  EXPECT_TOKEN(Tokens[3], tok::equal, TT_OverloadedOperator);
+  EXPECT_TOKEN(Tokens[4], tok::l_paren, TT_OverloadedOperatorLParen);
+  Tokens = annotate("x.operator+=()");
+  ASSERT_EQ(Tokens.size(), 7u) << Tokens;
+  EXPECT_TOKEN(Tokens[3], tok::plusequal, TT_OverloadedOperator);
+  EXPECT_TOKEN(Tokens[4], tok::l_paren, TT_OverloadedOperatorLParen);
+  Tokens = annotate("x.operator,()");
+  ASSERT_EQ(Tokens.size(), 7u) << Tokens;
+  EXPECT_TOKEN(Tokens[3], tok::comma, TT_OverloadedOperator);
+  EXPECT_TOKEN(Tokens[4], tok::l_paren, TT_OverloadedOperatorLParen);
+  Tokens = annotate("x.operator()()");
+  ASSERT_EQ(Tokens.size(), 8u) << Tokens;
+  EXPECT_TOKEN(Tokens[3], tok::l_paren, TT_OverloadedOperator);
+  EXPECT_TOKEN(Tokens[4], tok::r_paren, TT_OverloadedOperator);
+  EXPECT_TOKEN(Tokens[5], tok::l_paren, TT_OverloadedOperatorLParen);
+  Tokens = annotate("x.operator[]()");
+  ASSERT_EQ(Tokens.size(), 8u) << Tokens;
+  // EXPECT_TOKEN(Tokens[3], tok::l_square, TT_OverloadedOperator);
+  // EXPECT_TOKEN(Tokens[4], tok::r_square, TT_OverloadedOperator);
+  EXPECT_TOKEN(Tokens[5], tok::l_paren, TT_OverloadedOperatorLParen);
+  Tokens = annotate("x.operator\"\"_a()");
+  ASSERT_EQ(Tokens.size(), 7u) << Tokens;
+  EXPECT_TOKEN(Tokens[3], tok::string_literal, TT_OverloadedOperator);
+  EXPECT_TOKEN(Tokens[4], tok::l_paren, TT_OverloadedOperatorLParen);
+  Tokens = annotate("x.operator\"\"a()");
+  ASSERT_EQ(Tokens.size(), 8u) << Tokens;
+  // EXPECT_TOKEN(Tokens[3], tok::string_literal, TT_OverloadedOperator);
+  // EXPECT_TOKEN(Tokens[4], tok::identifier, TT_OverloadedOperator);
+  EXPECT_TOKEN(Tokens[5], tok::l_paren, TT_OverloadedOperatorLParen);
+}
+
 TEST_F(TokenAnnotatorTest, UnderstandsRequiresClausesAndConcepts) {
   auto Tokens = annotate("template <typename T>\n"
                          "concept C = (Foo && Bar) && (Bar && Baz);");
Index: clang/unittests/Format/FormatTest.cpp
===================================================================
--- clang/unittests/Format/FormatTest.cpp
+++ clang/unittests/Format/FormatTest.cpp
@@ -10149,6 +10149,11 @@
   // verifyFormat("void f() { operator*(a & a); }");
   // verifyFormat("void f() { operator&(a, b * b); }");
 
+  verifyFormat("void f() { return operator()(x) * b; }");
+  verifyFormat("void f() { return operator[](x) * b; }");
+  verifyFormat("void f() { return operator\"\"_a(x) * b; }");
+  // verifyFormat("void f() { return operator\"\"a(x) * b; }");
+
   verifyFormat("::operator delete(foo);");
   verifyFormat("::operator new(n * sizeof(foo));");
   verifyFormat("foo() { ::operator delete(foo); }");
Index: clang/lib/Format/TokenAnnotator.cpp
===================================================================
--- clang/lib/Format/TokenAnnotator.cpp
+++ clang/lib/Format/TokenAnnotator.cpp
@@ -1179,9 +1179,10 @@
             CurrentToken->Previous->isNot(tok::kw_operator)) {
           break;
         }
-        if (CurrentToken && CurrentToken->Previous->isOneOf(
-                                TT_BinaryOperator, TT_UnaryOperator, tok::comma,
-                                tok::star, tok::arrow, tok::amp, tok::ampamp)) {
+        if (CurrentToken &&
+            CurrentToken->Previous->isOneOf(
+                TT_BinaryOperator, TT_UnaryOperator, tok::comma, tok::star,
+                tok::arrow, tok::amp, tok::ampamp, tok::string_literal)) {
           CurrentToken->Previous->setType(TT_OverloadedOperator);
         }
       }


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D134853.464147.patch
Type: text/x-patch
Size: 4133 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20220930/69113e1c/attachment.bin>