[PATCH] D64634: [clangd] Fix duplicate highlighting tokens appearing in initializer lists

[Johan Vikström via Phabricator via cfe-commits]

jvikstrom created this revision.
jvikstrom added reviewers: hokein, sammccall, ilya-biryukov.
Herald added subscribers: cfe-commits, kadircet, arphaman, mgrang, jkorous, MaskRay.
Herald added a project: clang.

The RecursiveASTVisitor sometimes visits exprs in initializer lists twice. Added deduplication to prevent duplicate highlighting tokens from appearing. Done by sorting and a linear search.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D64634

Files:
  clang-tools-extra/clangd/SemanticHighlighting.cpp
  clang-tools-extra/clangd/SemanticHighlighting.h
  clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp


Index: clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp
===================================================================
--- clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp
+++ clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp
@@ -132,6 +132,13 @@
             $Namespace[[vwz]]::$Class[[A]]::$Enum[[B]]::Hi;
       ::$Namespace[[vwz]]::$Class[[A]] $Variable[[B]];
       ::$Namespace[[abc]]::$Namespace[[bcd]]::$Class[[A]] $Variable[[BB]];
+    )cpp",
+    R"cpp(
+      struct $Class[[AA]] {
+        int A;
+      }
+      int $Variable[[B]];
+      $Class[[AA]] $Variable[[A]]{$Variable[[B]]};
     )cpp"};
   for (const auto &TestCase : TestCases) {
     checkHighlightings(TestCase);
Index: clang-tools-extra/clangd/SemanticHighlighting.h
===================================================================
--- clang-tools-extra/clangd/SemanticHighlighting.h
+++ clang-tools-extra/clangd/SemanticHighlighting.h
@@ -40,6 +40,7 @@
 };
 
 bool operator==(const HighlightingToken &Lhs, const HighlightingToken &Rhs);
+bool operator<(const HighlightingToken &Lhs, const HighlightingToken &Rhs);
 
 // Returns all HighlightingTokens from an AST. Only generates highlights for the
 // main AST.
Index: clang-tools-extra/clangd/SemanticHighlighting.cpp
===================================================================
--- clang-tools-extra/clangd/SemanticHighlighting.cpp
+++ clang-tools-extra/clangd/SemanticHighlighting.cpp
@@ -31,6 +31,16 @@
   std::vector<HighlightingToken> collectTokens() {
     Tokens.clear();
     TraverseAST(Ctx);
+    // Initializer lists can give duplicates of tokens, therefore all tokens
+    // must be deduplicated.
+    std::sort(Tokens.begin(), Tokens.end());
+    for (unsigned I = 1; I < Tokens.size(); ++I) {
+      if (Tokens[I] == Tokens[I - 1]) {
+        Tokens.erase(Tokens.begin() + I);
+        --I;
+      }
+    }
+
     return Tokens;
   }
 
@@ -70,7 +80,6 @@
         DeclarationName::Identifier)
       // Only want to highlight identifiers.
       return true;
-
     addToken(Ref->getLocation(), Ref->getDecl());
     return true;
   }
@@ -201,6 +210,10 @@
   return Lhs.Kind == Rhs.Kind && Lhs.R == Rhs.R;
 }
 
+bool operator<(const HighlightingToken &Lhs, const HighlightingToken &Rhs) {
+  return Lhs.R.start < Rhs.R.start;
+}
+
 std::vector<HighlightingToken> getSemanticHighlightings(ParsedAST &AST) {
   return HighlightingTokenCollector(AST).collectTokens();
 }


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D64634.209479.patch
Type: text/x-patch
Size: 2473 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20190712/1a3518bc/attachment.bin>