[clang-tools-extra] r343778 - [clangd] clangd-indexer gathers refs and stores them in index files.

via cfe-commits cfe-commits at lists.llvm.org
Fri Oct 5 02:34:22 PDT 2018


Hi Sam,

Our internal build bot hit an assertion failure in the changes you made in this commit that I was able to reproduce by building your change on my machine on Windows 7 using Visual Studio 2015. None of the public build bots seem to have hit this issue, though I am not sure why at the moment.

Here is the assertion failure that is hit while running ClangdTests.exe:

[----------] 2 tests from SerializationTest
[ RUN      ] SerializationTest.YAMLConversions
[       OK ] SerializationTest.YAMLConversions (0 ms)
[ RUN      ] SerializationTest.BinaryConversions
Assertion failed: OutBufCur == OutBufStart && "raw_ostream destructor called with non-empty buffer!", file C:\src\upstream\llvm2\lib\Support\raw_ostream.cpp, line 73
0x000000013FA74405 (0x0000000000000016 0x0000000000000200 0x000006D100230006 0x000007FEEA9BAA51), HandleAbort() + 0x5 bytes(s), c:\src\upstream\llvm2\lib\support\windows\signals.inc, line 409
0x000007FEEAA1DC17 (0x0000000000000001 0x0000000100000000 0x0000000000000000 0x00000000009BF460), raise() + 0x1E7 bytes(s)
0x000007FEEAA1EAA1 (0x000007FE00000003 0x000007FE00000003 0x00000001413AF810 0x00000001413AF790), abort() + 0x31 bytes(s)
0x000007FEEAA20751 (0x0000000000000049 0x00000001413AF810 0x0000000000000122 0x000007FEEA9C05D6), _get_wpgmptr() + 0x1BE1 bytes(s)
0x000007FEEAA20A5F (0x00000000009BF890 0x00000000009BF680 0x0000000000B05AB0 0x000000014126F8C5), _wassert() + 0x3F bytes(s)
0x000000013FA40677 (0x0000000000000000 0x0000000000A80000 0x00000000009BF890 0x000000013FA41A2E), llvm::raw_ostream::~raw_ostream() + 0x37 bytes(s), c:\src\upstream\llvm2\lib\support\raw_ostream.cpp, line 75
0x000000013FA4057C (0x0000000004BA1FF0 0x00000000009BF680 0x0000000000B05AB0 0x0000000000B05AB0), llvm::raw_fd_ostream::~raw_fd_ostream() + 0x11C bytes(s), c:\src\upstream\llvm2\lib\support\raw_ostream.cpp, line 617
0x000000013F96A4CC (0x0000000004BA1FF0 0x0000000000B05AB0 0x0000000000B05AB0 0x0000000004BA1FF0), clang::clangd::`anonymous namespace'::SerializationTest_BinaryConversions_Test::TestBody() + 0x34C bytes(s), c:\src\upstream\llvm2\tools\clang\tools\extra\unittests\clangd\serializationtests.cpp, line 166
0x000000013FAE201C (0x0000000004BA1FF0 0x0000000000B05AB0 0x00000001413C4B68 0x000007FEEAA954B8), testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test,void>() + 0xC bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 2387 + 0x2 byte(s)
0x000000013FB09E5B (0x0000000004BA1FF0 0x0000000000B05AB0 0x00000001413C4D30 0x00000000009BF9E8), testing::Test::Run() + 0x7B bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 2481
0x000000013FB0A0DB (0x0000016641B94CAF 0x0000000000B05AB0 0x0000000000B1DE30 0x0000000000AFF560), testing::TestInfo::Run() + 0xAB bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 2660
0x000000013FB09F72 (0x0000000000000023 0x0000000000000000 0x0000000000B05AB0 0x0000000000AFF560), testing::TestCase::Run() + 0xB2 bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 2774 + 0x12 byte(s)
0x000000013FB0A529 (0x0000000000AFF410 0x0000000000000000 0x0000000000000001 0x000007FE00000001), testing::internal::UnitTestImpl::RunAllTests() + 0x229 bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 4649 + 0x39 byte(s)
0x000000013FAE213C (0x0000000000AFF410 0x0000000000000000 0x00000001413C54A8 0x0000000000000000), testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool>() + 0xC bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 2387 + 0x2 byte(s)
0x000000013FB0A27C (0x0000000000008002 0x0000000000000000 0x0000000000000000 0x0000000000000000), testing::UnitTest::Run() + 0xEC bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 4257 + 0x17 byte(s)
0x0000000141299397 (0x0000000100000001 0x0000000000000000 0x0000000000000000 0x01D45C4625BC735E), main() + 0xB7 bytes(s), c:\src\upstream\llvm2\utils\unittest\unittestmain\testmain.cpp, line 52
0x00000001412707C1 (0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000), __scrt_common_main_seh() + 0x11D bytes(s), f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl, line 253 + 0x22 byte(s)
0x0000000076C159CD (0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000), BaseThreadInitThunk() + 0xD bytes(s)
0x0000000076E7385D (0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000), RtlUserThreadStart() + 0x1D bytes(s)

Could you take a look into this?

Douglas Yung

> -----Original Message-----
> From: cfe-commits [mailto:cfe-commits-bounces at lists.llvm.org] On Behalf
> Of Sam McCall via cfe-commits
> Sent: Thursday, October 04, 2018 7:10
> To: cfe-commits at lists.llvm.org
> Subject: [clang-tools-extra] r343778 - [clangd] clangd-indexer gathers
> refs and stores them in index files.
> 
> Author: sammccall
> Date: Thu Oct  4 07:09:55 2018
> New Revision: 343778
> 
> URL: http://llvm.org/viewvc/llvm-project?rev=343778&view=rev
> Log:
> [clangd] clangd-indexer gathers refs and stores them in index files.
> 
> Reviewers: ioeric
> 
> Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-
> commits
> 
> Differential Revision: https://reviews.llvm.org/D52531
> 
> Modified:
>     clang-tools-extra/trunk/clangd/index/IndexAction.cpp
>     clang-tools-extra/trunk/clangd/index/IndexAction.h
>     clang-tools-extra/trunk/clangd/index/Serialization.cpp
>     clang-tools-extra/trunk/clangd/index/Serialization.h
>     clang-tools-extra/trunk/clangd/index/YAMLSerialization.cpp
>     clang-tools-extra/trunk/clangd/indexer/IndexerMain.cpp
>     clang-tools-extra/trunk/unittests/clangd/SerializationTests.cpp
> 
> Modified: clang-tools-extra/trunk/clangd/index/IndexAction.cpp
> URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> extra/trunk/clangd/index/IndexAction.cpp?rev=343778&r1=343777&r2=343778
> &view=diff
> =======================================================================
> =======
> --- clang-tools-extra/trunk/clangd/index/IndexAction.cpp (original)
> +++ clang-tools-extra/trunk/clangd/index/IndexAction.cpp Thu Oct  4
> 07:09:55 2018
> @@ -13,10 +13,11 @@ public:
>    IndexAction(std::shared_ptr<SymbolCollector> C,
>                std::unique_ptr<CanonicalIncludes> Includes,
>                const index::IndexingOptions &Opts,
> -              std::function<void(SymbolSlab)> &SymbolsCallback)
> +              std::function<void(SymbolSlab)> SymbolsCallback,
> +              std::function<void(RefSlab)> RefsCallback)
>        : WrapperFrontendAction(index::createIndexingAction(C, Opts,
> nullptr)),
> -        SymbolsCallback(SymbolsCallback), Collector(C),
> -        Includes(std::move(Includes)),
> +        SymbolsCallback(SymbolsCallback), RefsCallback(RefsCallback),
> +        Collector(C), Includes(std::move(Includes)),
>          PragmaHandler(collectIWYUHeaderMaps(this->Includes.get())) {}
> 
>    std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
> @@ -41,10 +42,13 @@ public:
>        return;
>      }
>      SymbolsCallback(Collector->takeSymbols());
> +    if (RefsCallback != nullptr)
> +      RefsCallback(Collector->takeRefs());
>    }
> 
>  private:
>    std::function<void(SymbolSlab)> SymbolsCallback;
> +  std::function<void(RefSlab)> RefsCallback;
>    std::shared_ptr<SymbolCollector> Collector;
>    std::unique_ptr<CanonicalIncludes> Includes;
>    std::unique_ptr<CommentHandler> PragmaHandler;
> @@ -54,20 +58,23 @@ private:
> 
>  std::unique_ptr<FrontendAction>
>  createStaticIndexingAction(SymbolCollector::Options Opts,
> -                           std::function<void(SymbolSlab)>
> SymbolsCallback) {
> +                           std::function<void(SymbolSlab)>
> SymbolsCallback,
> +                           std::function<void(RefSlab)> RefsCallback)
> {
>    index::IndexingOptions IndexOpts;
>    IndexOpts.SystemSymbolFilter =
>        index::IndexingOptions::SystemSymbolFilterKind::All;
>    Opts.CollectIncludePath = true;
>    Opts.CountReferences = true;
>    Opts.Origin = SymbolOrigin::Static;
> +  if (RefsCallback != nullptr)
> +    Opts.RefFilter = RefKind::All;
>    auto Includes = llvm::make_unique<CanonicalIncludes>();
>    addSystemHeadersMapping(Includes.get());
>    Opts.Includes = Includes.get();
>    return llvm::make_unique<IndexAction>(
>        std::make_shared<SymbolCollector>(std::move(Opts)),
> std::move(Includes),
> -      IndexOpts, SymbolsCallback);
> -}
> +      IndexOpts, SymbolsCallback, RefsCallback);
> +};
> 
>  } // namespace clangd
>  } // namespace clang
> 
> Modified: clang-tools-extra/trunk/clangd/index/IndexAction.h
> URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> extra/trunk/clangd/index/IndexAction.h?rev=343778&r1=343777&r2=343778&v
> iew=diff
> =======================================================================
> =======
> --- clang-tools-extra/trunk/clangd/index/IndexAction.h (original)
> +++ clang-tools-extra/trunk/clangd/index/IndexAction.h Thu Oct  4
> 07:09:55 2018
> @@ -21,10 +21,13 @@ namespace clangd {
>  // Only a subset of SymbolCollector::Options are respected:
>  //   - include paths are always collected, and canonicalized
> appropriately
>  //   - references are always counted
> +//   - main-file refs are collected (if RefsCallback is non-null)
>  //   - the symbol origin is always Static
> +// FIXME: refs from headers should also be collected.
>  std::unique_ptr<FrontendAction>
>  createStaticIndexingAction(SymbolCollector::Options Opts,
> -                           std::function<void(SymbolSlab)>
> SymbolsCallback);
> +                           std::function<void(SymbolSlab)>
> SymbolsCallback,
> +                           std::function<void(RefSlab)> RefsCallback);
> 
>  } // namespace clangd
>  } // namespace clang
> 
> Modified: clang-tools-extra/trunk/clangd/index/Serialization.cpp
> URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> extra/trunk/clangd/index/Serialization.cpp?rev=343778&r1=343777&r2=3437
> 78&view=diff
> =======================================================================
> =======
> --- clang-tools-extra/trunk/clangd/index/Serialization.cpp (original)
> +++ clang-tools-extra/trunk/clangd/index/Serialization.cpp Thu Oct  4
> 07:09:55 2018
> @@ -298,17 +298,47 @@ Symbol readSymbol(Reader &Data, ArrayRef
>    return Sym;
>  }
> 
> +// REFS ENCODING
> +// A refs section has data grouped by Symbol. Each symbol has:
> +//  - SymbolID: 20 bytes
> +//  - NumRefs: varint
> +//  - Ref[NumRefs]
> +// Fields of Ref are encoded in turn, see implementation.
> +
> +void writeRefs(const SymbolID &ID, ArrayRef<Ref> Refs,
> +               const StringTableOut &Strings, raw_ostream &OS) {
> +  OS << ID.raw();
> +  writeVar(Refs.size(), OS);
> +  for (const auto &Ref : Refs) {
> +    OS.write(static_cast<unsigned char>(Ref.Kind));
> +    writeLocation(Ref.Location, Strings, OS);
> +  }
> +}
> +
> +std::pair<SymbolID, std::vector<Ref>> readRefs(Reader &Data,
> +                                               ArrayRef<StringRef>
> Strings) {
> +  std::pair<SymbolID, std::vector<Ref>> Result;
> +  Result.first = Data.consumeID();
> +  Result.second.resize(Data.consumeVar());
> +  for (auto &Ref : Result.second) {
> +    Ref.Kind = static_cast<RefKind>(Data.consume8());
> +    Ref.Location = readLocation(Data, Strings);
> +  }
> +  return Result;
> +}
> +
>  // FILE ENCODING
>  // A file is a RIFF chunk with type 'CdIx'.
>  // It contains the sections:
>  //   - meta: version number
>  //   - stri: string table
>  //   - symb: symbols
> +//   - refs: references to symbols
> 
>  // The current versioning scheme is simple - non-current versions are
> rejected.
>  // If you make a breaking change, bump this version number to
> invalidate stored
>  // data. Later we may want to support some backward compatibility.
> -constexpr static uint32_t Version = 4;
> +constexpr static uint32_t Version = 5;
> 
>  Expected<IndexFileIn> readRIFF(StringRef Data) {
>    auto RIFF = riff::readFile(Data);
> @@ -342,6 +372,18 @@ Expected<IndexFileIn> readRIFF(StringRef
>        return makeError("malformed or truncated symbol");
>      Result.Symbols = std::move(Symbols).build();
>    }
> +  if (Chunks.count("refs")) {
> +    Reader RefsReader(Chunks.lookup("refs"));
> +    RefSlab::Builder Refs;
> +    while (!RefsReader.eof()) {
> +      auto RefsBundle = readRefs(RefsReader, Strings->Strings);
> +      for (const auto &Ref : RefsBundle.second) // FIXME: bulk insert?
> +        Refs.insert(RefsBundle.first, Ref);
> +    }
> +    if (RefsReader.err())
> +      return makeError("malformed or truncated refs");
> +    Result.Refs = std::move(Refs).build();
> +  }
>    return std::move(Result);
>  }
> 
> @@ -363,6 +405,14 @@ void writeRIFF(const IndexFileOut &Data,
>      Symbols.emplace_back(Sym);
>      visitStrings(Symbols.back(), [&](StringRef &S) {
> Strings.intern(S); });
>    }
> +  std::vector<std::pair<SymbolID, std::vector<Ref>>> Refs;
> +  if (Data.Refs) {
> +    for (const auto &Sym : *Data.Refs) {
> +      Refs.emplace_back(Sym);
> +      for (auto &Ref : Refs.back().second)
> +        Strings.intern(Ref.Location.FileURI);
> +    }
> +  }
> 
>    std::string StringSection;
>    {
> @@ -379,6 +429,16 @@ void writeRIFF(const IndexFileOut &Data,
>    }
>    RIFF.Chunks.push_back({riff::fourCC("symb"), SymbolSection});
> 
> +  std::string RefsSection;
> +  if (Data.Refs) {
> +    {
> +      raw_string_ostream RefsOS(RefsSection);
> +      for (const auto &Sym : Refs)
> +        writeRefs(Sym.first, Sym.second, Strings, RefsOS);
> +    }
> +    RIFF.Chunks.push_back({riff::fourCC("refs"), RefsSection});
> +  }
> +
>    OS << RIFF;
>  }
> 
> @@ -428,6 +488,8 @@ std::unique_ptr<SymbolIndex> loadIndex(l
>      if (auto I = readIndexFile(Buffer->get()->getBuffer())) {
>        if (I->Symbols)
>          Symbols = std::move(*I->Symbols);
> +      if (I->Refs)
> +        Refs = std::move(*I->Refs);
>      } else {
>        llvm::errs() << "Bad Index: " << llvm::toString(I.takeError())
> << "\n";
>        return nullptr;
> 
> Modified: clang-tools-extra/trunk/clangd/index/Serialization.h
> URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> extra/trunk/clangd/index/Serialization.h?rev=343778&r1=343777&r2=343778
> &view=diff
> =======================================================================
> =======
> --- clang-tools-extra/trunk/clangd/index/Serialization.h (original)
> +++ clang-tools-extra/trunk/clangd/index/Serialization.h Thu Oct  4
> 07:09:55 2018
> @@ -38,26 +38,29 @@ enum class IndexFileFormat {
>  // Holds the contents of an index file that was read.
>  struct IndexFileIn {
>    llvm::Optional<SymbolSlab> Symbols;
> +  llvm::Optional<RefSlab> Refs;
>  };
> -// Parse an index file. The input must be a RIFF container chunk.
> +// Parse an index file. The input must be a RIFF or YAML file.
>  llvm::Expected<IndexFileIn> readIndexFile(llvm::StringRef);
> 
>  // Specifies the contents of an index file to be written.
>  struct IndexFileOut {
> -  const SymbolSlab *Symbols;
> -  // TODO: Support serializing symbol occurrences.
> +  const SymbolSlab *Symbols = nullptr;
> +  const RefSlab *Refs = nullptr;
>    // TODO: Support serializing Dex posting lists.
>    IndexFileFormat Format = IndexFileFormat::RIFF;
> 
>    IndexFileOut() = default;
>    IndexFileOut(const IndexFileIn &I)
> -      : Symbols(I.Symbols ? I.Symbols.getPointer() : nullptr) {}
> +      : Symbols(I.Symbols ? I.Symbols.getPointer() : nullptr),
> +        Refs(I.Refs ? I.Refs.getPointer() : nullptr) {}
>  };
>  // Serializes an index file.
>  llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, const
> IndexFileOut &O);
> 
>  // Convert a single symbol to YAML, a nice debug representation.
>  std::string toYAML(const Symbol &);
> +std::string toYAML(const std::pair<SymbolID, ArrayRef<Ref>> &);
> 
>  // Build an in-memory static index from an index file.
>  // The size should be relatively small, so data can be managed in
> memory.
> 
> Modified: clang-tools-extra/trunk/clangd/index/YAMLSerialization.cpp
> URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> extra/trunk/clangd/index/YAMLSerialization.cpp?rev=343778&r1=343777&r2=
> 343778&view=diff
> =======================================================================
> =======
> --- clang-tools-extra/trunk/clangd/index/YAMLSerialization.cpp
> (original)
> +++ clang-tools-extra/trunk/clangd/index/YAMLSerialization.cpp Thu Oct
> 4 07:09:55 2018
> @@ -6,6 +6,12 @@
>  // License. See LICENSE.TXT for details.
>  //
>  //===-----------------------------------------------------------------
> -----===//
> +//
> +// A YAML index file is a sequence of tagged entries.
> +// Each entry either encodes a Symbol or the list of references to a
> symbol
> +// (a "ref bundle").
> +//
> +//===-----------------------------------------------------------------
> -----===//
> 
>  #include "Index.h"
>  #include "Serialization.h"
> @@ -20,10 +26,22 @@
>  #include <cstdint>
> 
> 
> LLVM_YAML_IS_SEQUENCE_VECTOR(clang::clangd::Symbol::IncludeHeaderWithRe
> ferences)
> +LLVM_YAML_IS_SEQUENCE_VECTOR(clang::clangd::Ref)
> 
> +namespace {
> +using RefBundle =
> +    std::pair<clang::clangd::SymbolID,
> std::vector<clang::clangd::Ref>>;
> +// This is a pale imitation of std::variant<Symbol, RefBundle>
> +struct VariantEntry {
> +  llvm::Optional<clang::clangd::Symbol> Symbol;
> +  llvm::Optional<RefBundle> Refs;
> +};
> +} // namespace
>  namespace llvm {
>  namespace yaml {
> 
> +using clang::clangd::Ref;
> +using clang::clangd::RefKind;
>  using clang::clangd::Symbol;
>  using clang::clangd::SymbolID;
>  using clang::clangd::SymbolLocation;
> @@ -179,6 +197,46 @@ template <> struct ScalarEnumerationTrai
>    }
>  };
> 
> +template <> struct MappingTraits<RefBundle> {
> +  static void mapping(IO &IO, RefBundle &Refs) {
> +    MappingNormalization<NormalizedSymbolID, SymbolID> NSymbolID(IO,
> +
> Refs.first);
> +    IO.mapRequired("ID", NSymbolID->HexString);
> +    IO.mapRequired("References", Refs.second);
> +  }
> +};
> +
> +struct NormalizedRefKind {
> +  NormalizedRefKind(IO &) {}
> +  NormalizedRefKind(IO &, RefKind O) { Kind = static_cast<uint8_t>(O);
> }
> +
> +  RefKind denormalize(IO &) { return static_cast<RefKind>(Kind); }
> +
> +  uint8_t Kind = 0;
> +};
> +
> +template <> struct MappingTraits<Ref> {
> +  static void mapping(IO &IO, Ref &R) {
> +    MappingNormalization<NormalizedRefKind, RefKind> NKind(IO,
> R.Kind);
> +    IO.mapRequired("Kind", NKind->Kind);
> +    IO.mapRequired("Location", R.Location);
> +  }
> +};
> +
> +template <> struct MappingTraits<VariantEntry> {
> +  static void mapping(IO &IO, VariantEntry &Variant) {
> +    if (IO.mapTag("!Symbol", Variant.Symbol.hasValue())) {
> +      if (!IO.outputting())
> +        Variant.Symbol.emplace();
> +      MappingTraits<Symbol>::mapping(IO, *Variant.Symbol);
> +    } else if (IO.mapTag("!Refs", Variant.Refs.hasValue())) {
> +      if (!IO.outputting())
> +        Variant.Refs.emplace();
> +      MappingTraits<RefBundle>::mapping(IO, *Variant.Refs);
> +    }
> +  }
> +};
> +
>  } // namespace yaml
>  } // namespace llvm
> 
> @@ -187,23 +245,38 @@ namespace clangd {
> 
>  void writeYAML(const IndexFileOut &O, raw_ostream &OS) {
>    llvm::yaml::Output Yout(OS);
> -  for (Symbol Sym : *O.Symbols) // copy: Yout<< requires mutability.
> -    Yout << Sym;
> +  for (const auto &Sym : *O.Symbols) {
> +    VariantEntry Entry;
> +    Entry.Symbol = Sym;
> +    Yout << Entry;
> +  }
> +  if (O.Refs)
> +    for (auto &Sym : *O.Refs) {
> +      VariantEntry Entry;
> +      Entry.Refs = Sym;
> +      Yout << Entry;
> +    }
>  }
> 
>  Expected<IndexFileIn> readYAML(StringRef Data) {
>    SymbolSlab::Builder Symbols;
> +  RefSlab::Builder Refs;
>    llvm::yaml::Input Yin(Data);
>    do {
> -    Symbol S;
> -    Yin >> S;
> +    VariantEntry Variant;
> +    Yin >> Variant;
>      if (Yin.error())
>        return llvm::errorCodeToError(Yin.error());
> -    Symbols.insert(S);
> +    if (Variant.Symbol)
> +      Symbols.insert(*Variant.Symbol);
> +    if (Variant.Refs)
> +      for (const auto &Ref : Variant.Refs->second)
> +        Refs.insert(Variant.Refs->first, Ref);
>    } while (Yin.nextDocument());
> 
>    IndexFileIn Result;
>    Result.Symbols.emplace(std::move(Symbols).build());
> +  Result.Refs.emplace(std::move(Refs).build());
>    return std::move(Result);
>  }
> 
> @@ -217,6 +290,17 @@ std::string toYAML(const Symbol &S) {
>    }
>    return Buf;
>  }
> +
> +std::string toYAML(const std::pair<SymbolID, ArrayRef<Ref>> &Data) {
> +  RefBundle Refs = {Data.first, Data.second};
> +  std::string Buf;
> +  {
> +    llvm::raw_string_ostream OS(Buf);
> +    llvm::yaml::Output Yout(OS);
> +    Yout << Refs;
> +  }
> +  return Buf;
> +}
> 
>  } // namespace clangd
>  } // namespace clang
> 
> Modified: clang-tools-extra/trunk/clangd/indexer/IndexerMain.cpp
> URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> extra/trunk/clangd/indexer/IndexerMain.cpp?rev=343778&r1=343777&r2=3437
> 78&view=diff
> =======================================================================
> =======
> --- clang-tools-extra/trunk/clangd/indexer/IndexerMain.cpp (original)
> +++ clang-tools-extra/trunk/clangd/indexer/IndexerMain.cpp Thu Oct  4
> 07:09:55 2018
> @@ -67,18 +67,30 @@ public:
>                     else
>                       Symbols.insert(Sym);
>                   }
> +               },
> +               [&](RefSlab S) {
> +                 std::lock_guard<std::mutex> Lock(SymbolsMu);
> +                 for (const auto &Sym : S) {
> +                   // No need to merge as currently all Refs are from
> main file.
> +                   for (const auto &Ref : Sym.second)
> +                     Refs.insert(Sym.first, Ref);
> +                 }
>                 })
>          .release();
>    }
> 
>    // Awkward: we write the result in the destructor, because the
> executor
>    // takes ownership so it's the easiest way to get our data back out.
> -  ~IndexActionFactory() { Result.Symbols = std::move(Symbols).build();
> }
> +  ~IndexActionFactory() {
> +    Result.Symbols = std::move(Symbols).build();
> +    Result.Refs = std::move(Refs).build();
> +  }
> 
>  private:
>    IndexFileIn &Result;
>    std::mutex SymbolsMu;
>    SymbolSlab::Builder Symbols;
> +  RefSlab::Builder Refs;
>  };
> 
>  } // namespace
> 
> Modified: clang-tools-
> extra/trunk/unittests/clangd/SerializationTests.cpp
> URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> extra/trunk/unittests/clangd/SerializationTests.cpp?rev=343778&r1=34377
> 7&r2=343778&view=diff
> =======================================================================
> =======
> --- clang-tools-extra/trunk/unittests/clangd/SerializationTests.cpp
> (original)
> +++ clang-tools-extra/trunk/unittests/clangd/SerializationTests.cpp Thu
> Oct  4 07:09:55 2018
> @@ -13,6 +13,9 @@
>  #include "gmock/gmock.h"
>  #include "gtest/gtest.h"
> 
> +using testing::_;
> +using testing::AllOf;
> +using testing::Pair;
>  using testing::UnorderedElementsAre;
>  using testing::UnorderedElementsAreArray;
>  namespace clang {
> @@ -21,6 +24,7 @@ namespace {
> 
>  const char *YAML = R"(
>  ---
> +!Symbol
>  ID: 057557CEBF6E6B2DD437FBF60CC58F352D1DF856
>  Name:   'Foo1'
>  Scope:   'clang::'
> @@ -46,6 +50,7 @@ IncludeHeaders:
>      References:    3
>  ...
>  ---
> +!Symbol
>  ID: 057557CEBF6E6B2DD437FBF60CC58F352D1DF858
>  Name:   'Foo2'
>  Scope:   'clang::'
> @@ -64,6 +69,18 @@ Flags:    2
>  Signature:    '-sig'
>  CompletionSnippetSuffix:    '-snippet'
>  ...
> +!Refs
> +ID: 057557CEBF6E6B2DD437FBF60CC58F352D1DF856
> +References:
> +  - Kind: 4
> +    Location:
> +      FileURI:    file:///path/foo.cc
> +      Start:
> +        Line: 5
> +        Column: 3
> +      End:
> +        Line: 5
> +        Column: 8
>  )";
> 
>  MATCHER_P(ID, I, "") { return arg.ID ==
> cantFail(SymbolID::fromStr(I)); }
> @@ -107,6 +124,16 @@ TEST(SerializationTest, YAMLConversions)
>    EXPECT_EQ(Sym2.CanonicalDeclaration.FileURI, "file:///path/bar.h");
>    EXPECT_FALSE(Sym2.Flags & Symbol::IndexedForCodeCompletion);
>    EXPECT_TRUE(Sym2.Flags & Symbol::Deprecated);
> +
> +  ASSERT_TRUE(bool(ParsedYAML->Refs));
> +  EXPECT_THAT(*ParsedYAML->Refs,
> +              UnorderedElementsAre(
> +                  Pair(cantFail(SymbolID::fromStr(
> +
> "057557CEBF6E6B2DD437FBF60CC58F352D1DF856")),
> +                       testing::SizeIs(1))));
> +  auto Ref1 = ParsedYAML->Refs->begin()->second.front();
> +  EXPECT_EQ(Ref1.Kind, RefKind::Reference);
> +  EXPECT_EQ(Ref1.Location.FileURI, "file:///path/foo.cc");
>  }
> 
>  std::vector<std::string> YAMLFromSymbols(const SymbolSlab &Slab) {
> @@ -115,24 +142,37 @@ std::vector<std::string> YAMLFromSymbols
>      Result.push_back(toYAML(Sym));
>    return Result;
>  }
> +std::vector<std::string> YAMLFromRefs(const RefSlab &Slab) {
> +  std::vector<std::string> Result;
> +  for (const auto &Sym : Slab)
> +    Result.push_back(toYAML(Sym));
> +  return Result;
> +}
> 
>  TEST(SerializationTest, BinaryConversions) {
>    auto In = readIndexFile(YAML);
>    EXPECT_TRUE(bool(In)) << In.takeError();
> 
>    // Write to binary format, and parse again.
> -  IndexFileOut Out;
> -  Out.Symbols = In->Symbols.getPointer();
> +  IndexFileOut Out(*In);
>    Out.Format = IndexFileFormat::RIFF;
>    std::string Serialized = llvm::to_string(Out);
> +  {
> +    std::error_code EC;
> +    llvm::raw_fd_ostream F("/tmp/foo", EC);
> +    F << Serialized;
> +  }
> 
>    auto In2 = readIndexFile(Serialized);
>    ASSERT_TRUE(bool(In2)) << In.takeError();
> -  ASSERT_TRUE(In->Symbols);
> +  ASSERT_TRUE(In2->Symbols);
> +  ASSERT_TRUE(In2->Refs);
> 
>    // Assert the YAML serializations match, for nice comparisons and
> diffs.
>    EXPECT_THAT(YAMLFromSymbols(*In2->Symbols),
>                UnorderedElementsAreArray(YAMLFromSymbols(*In-
> >Symbols)));
> +  EXPECT_THAT(YAMLFromRefs(*In2->Refs),
> +              UnorderedElementsAreArray(YAMLFromRefs(*In->Refs)));
>  }
> 
>  } // namespace
> 
> 
> _______________________________________________
> cfe-commits mailing list
> cfe-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits


More information about the cfe-commits mailing list