[clang-tools-extra] r343778 - [clangd] clangd-indexer gathers refs and stores them in index files.
via cfe-commits
cfe-commits at lists.llvm.org
Fri Oct 5 02:34:22 PDT 2018
Hi Sam,
Our internal build bot hit an assertion failure in the changes you made in this commit that I was able to reproduce by building your change on my machine on Windows 7 using Visual Studio 2015. None of the public build bots seem to have hit this issue, though I am not sure why at the moment.
Here is the assertion failure that is hit while running ClangdTests.exe:
[----------] 2 tests from SerializationTest
[ RUN ] SerializationTest.YAMLConversions
[ OK ] SerializationTest.YAMLConversions (0 ms)
[ RUN ] SerializationTest.BinaryConversions
Assertion failed: OutBufCur == OutBufStart && "raw_ostream destructor called with non-empty buffer!", file C:\src\upstream\llvm2\lib\Support\raw_ostream.cpp, line 73
0x000000013FA74405 (0x0000000000000016 0x0000000000000200 0x000006D100230006 0x000007FEEA9BAA51), HandleAbort() + 0x5 bytes(s), c:\src\upstream\llvm2\lib\support\windows\signals.inc, line 409
0x000007FEEAA1DC17 (0x0000000000000001 0x0000000100000000 0x0000000000000000 0x00000000009BF460), raise() + 0x1E7 bytes(s)
0x000007FEEAA1EAA1 (0x000007FE00000003 0x000007FE00000003 0x00000001413AF810 0x00000001413AF790), abort() + 0x31 bytes(s)
0x000007FEEAA20751 (0x0000000000000049 0x00000001413AF810 0x0000000000000122 0x000007FEEA9C05D6), _get_wpgmptr() + 0x1BE1 bytes(s)
0x000007FEEAA20A5F (0x00000000009BF890 0x00000000009BF680 0x0000000000B05AB0 0x000000014126F8C5), _wassert() + 0x3F bytes(s)
0x000000013FA40677 (0x0000000000000000 0x0000000000A80000 0x00000000009BF890 0x000000013FA41A2E), llvm::raw_ostream::~raw_ostream() + 0x37 bytes(s), c:\src\upstream\llvm2\lib\support\raw_ostream.cpp, line 75
0x000000013FA4057C (0x0000000004BA1FF0 0x00000000009BF680 0x0000000000B05AB0 0x0000000000B05AB0), llvm::raw_fd_ostream::~raw_fd_ostream() + 0x11C bytes(s), c:\src\upstream\llvm2\lib\support\raw_ostream.cpp, line 617
0x000000013F96A4CC (0x0000000004BA1FF0 0x0000000000B05AB0 0x0000000000B05AB0 0x0000000004BA1FF0), clang::clangd::`anonymous namespace'::SerializationTest_BinaryConversions_Test::TestBody() + 0x34C bytes(s), c:\src\upstream\llvm2\tools\clang\tools\extra\unittests\clangd\serializationtests.cpp, line 166
0x000000013FAE201C (0x0000000004BA1FF0 0x0000000000B05AB0 0x00000001413C4B68 0x000007FEEAA954B8), testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test,void>() + 0xC bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 2387 + 0x2 byte(s)
0x000000013FB09E5B (0x0000000004BA1FF0 0x0000000000B05AB0 0x00000001413C4D30 0x00000000009BF9E8), testing::Test::Run() + 0x7B bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 2481
0x000000013FB0A0DB (0x0000016641B94CAF 0x0000000000B05AB0 0x0000000000B1DE30 0x0000000000AFF560), testing::TestInfo::Run() + 0xAB bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 2660
0x000000013FB09F72 (0x0000000000000023 0x0000000000000000 0x0000000000B05AB0 0x0000000000AFF560), testing::TestCase::Run() + 0xB2 bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 2774 + 0x12 byte(s)
0x000000013FB0A529 (0x0000000000AFF410 0x0000000000000000 0x0000000000000001 0x000007FE00000001), testing::internal::UnitTestImpl::RunAllTests() + 0x229 bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 4649 + 0x39 byte(s)
0x000000013FAE213C (0x0000000000AFF410 0x0000000000000000 0x00000001413C54A8 0x0000000000000000), testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool>() + 0xC bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 2387 + 0x2 byte(s)
0x000000013FB0A27C (0x0000000000008002 0x0000000000000000 0x0000000000000000 0x0000000000000000), testing::UnitTest::Run() + 0xEC bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 4257 + 0x17 byte(s)
0x0000000141299397 (0x0000000100000001 0x0000000000000000 0x0000000000000000 0x01D45C4625BC735E), main() + 0xB7 bytes(s), c:\src\upstream\llvm2\utils\unittest\unittestmain\testmain.cpp, line 52
0x00000001412707C1 (0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000), __scrt_common_main_seh() + 0x11D bytes(s), f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl, line 253 + 0x22 byte(s)
0x0000000076C159CD (0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000), BaseThreadInitThunk() + 0xD bytes(s)
0x0000000076E7385D (0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000), RtlUserThreadStart() + 0x1D bytes(s)
Could you take a look into this?
Douglas Yung
> -----Original Message-----
> From: cfe-commits [mailto:cfe-commits-bounces at lists.llvm.org] On Behalf
> Of Sam McCall via cfe-commits
> Sent: Thursday, October 04, 2018 7:10
> To: cfe-commits at lists.llvm.org
> Subject: [clang-tools-extra] r343778 - [clangd] clangd-indexer gathers
> refs and stores them in index files.
>
> Author: sammccall
> Date: Thu Oct 4 07:09:55 2018
> New Revision: 343778
>
> URL: http://llvm.org/viewvc/llvm-project?rev=343778&view=rev
> Log:
> [clangd] clangd-indexer gathers refs and stores them in index files.
>
> Reviewers: ioeric
>
> Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-
> commits
>
> Differential Revision: https://reviews.llvm.org/D52531
>
> Modified:
> clang-tools-extra/trunk/clangd/index/IndexAction.cpp
> clang-tools-extra/trunk/clangd/index/IndexAction.h
> clang-tools-extra/trunk/clangd/index/Serialization.cpp
> clang-tools-extra/trunk/clangd/index/Serialization.h
> clang-tools-extra/trunk/clangd/index/YAMLSerialization.cpp
> clang-tools-extra/trunk/clangd/indexer/IndexerMain.cpp
> clang-tools-extra/trunk/unittests/clangd/SerializationTests.cpp
>
> Modified: clang-tools-extra/trunk/clangd/index/IndexAction.cpp
> URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> extra/trunk/clangd/index/IndexAction.cpp?rev=343778&r1=343777&r2=343778
> &view=diff
> =======================================================================
> =======
> --- clang-tools-extra/trunk/clangd/index/IndexAction.cpp (original)
> +++ clang-tools-extra/trunk/clangd/index/IndexAction.cpp Thu Oct 4
> 07:09:55 2018
> @@ -13,10 +13,11 @@ public:
> IndexAction(std::shared_ptr<SymbolCollector> C,
> std::unique_ptr<CanonicalIncludes> Includes,
> const index::IndexingOptions &Opts,
> - std::function<void(SymbolSlab)> &SymbolsCallback)
> + std::function<void(SymbolSlab)> SymbolsCallback,
> + std::function<void(RefSlab)> RefsCallback)
> : WrapperFrontendAction(index::createIndexingAction(C, Opts,
> nullptr)),
> - SymbolsCallback(SymbolsCallback), Collector(C),
> - Includes(std::move(Includes)),
> + SymbolsCallback(SymbolsCallback), RefsCallback(RefsCallback),
> + Collector(C), Includes(std::move(Includes)),
> PragmaHandler(collectIWYUHeaderMaps(this->Includes.get())) {}
>
> std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
> @@ -41,10 +42,13 @@ public:
> return;
> }
> SymbolsCallback(Collector->takeSymbols());
> + if (RefsCallback != nullptr)
> + RefsCallback(Collector->takeRefs());
> }
>
> private:
> std::function<void(SymbolSlab)> SymbolsCallback;
> + std::function<void(RefSlab)> RefsCallback;
> std::shared_ptr<SymbolCollector> Collector;
> std::unique_ptr<CanonicalIncludes> Includes;
> std::unique_ptr<CommentHandler> PragmaHandler;
> @@ -54,20 +58,23 @@ private:
>
> std::unique_ptr<FrontendAction>
> createStaticIndexingAction(SymbolCollector::Options Opts,
> - std::function<void(SymbolSlab)>
> SymbolsCallback) {
> + std::function<void(SymbolSlab)>
> SymbolsCallback,
> + std::function<void(RefSlab)> RefsCallback)
> {
> index::IndexingOptions IndexOpts;
> IndexOpts.SystemSymbolFilter =
> index::IndexingOptions::SystemSymbolFilterKind::All;
> Opts.CollectIncludePath = true;
> Opts.CountReferences = true;
> Opts.Origin = SymbolOrigin::Static;
> + if (RefsCallback != nullptr)
> + Opts.RefFilter = RefKind::All;
> auto Includes = llvm::make_unique<CanonicalIncludes>();
> addSystemHeadersMapping(Includes.get());
> Opts.Includes = Includes.get();
> return llvm::make_unique<IndexAction>(
> std::make_shared<SymbolCollector>(std::move(Opts)),
> std::move(Includes),
> - IndexOpts, SymbolsCallback);
> -}
> + IndexOpts, SymbolsCallback, RefsCallback);
> +};
>
> } // namespace clangd
> } // namespace clang
>
> Modified: clang-tools-extra/trunk/clangd/index/IndexAction.h
> URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> extra/trunk/clangd/index/IndexAction.h?rev=343778&r1=343777&r2=343778&v
> iew=diff
> =======================================================================
> =======
> --- clang-tools-extra/trunk/clangd/index/IndexAction.h (original)
> +++ clang-tools-extra/trunk/clangd/index/IndexAction.h Thu Oct 4
> 07:09:55 2018
> @@ -21,10 +21,13 @@ namespace clangd {
> // Only a subset of SymbolCollector::Options are respected:
> // - include paths are always collected, and canonicalized
> appropriately
> // - references are always counted
> +// - main-file refs are collected (if RefsCallback is non-null)
> // - the symbol origin is always Static
> +// FIXME: refs from headers should also be collected.
> std::unique_ptr<FrontendAction>
> createStaticIndexingAction(SymbolCollector::Options Opts,
> - std::function<void(SymbolSlab)>
> SymbolsCallback);
> + std::function<void(SymbolSlab)>
> SymbolsCallback,
> + std::function<void(RefSlab)> RefsCallback);
>
> } // namespace clangd
> } // namespace clang
>
> Modified: clang-tools-extra/trunk/clangd/index/Serialization.cpp
> URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> extra/trunk/clangd/index/Serialization.cpp?rev=343778&r1=343777&r2=3437
> 78&view=diff
> =======================================================================
> =======
> --- clang-tools-extra/trunk/clangd/index/Serialization.cpp (original)
> +++ clang-tools-extra/trunk/clangd/index/Serialization.cpp Thu Oct 4
> 07:09:55 2018
> @@ -298,17 +298,47 @@ Symbol readSymbol(Reader &Data, ArrayRef
> return Sym;
> }
>
> +// REFS ENCODING
> +// A refs section has data grouped by Symbol. Each symbol has:
> +// - SymbolID: 20 bytes
> +// - NumRefs: varint
> +// - Ref[NumRefs]
> +// Fields of Ref are encoded in turn, see implementation.
> +
> +void writeRefs(const SymbolID &ID, ArrayRef<Ref> Refs,
> + const StringTableOut &Strings, raw_ostream &OS) {
> + OS << ID.raw();
> + writeVar(Refs.size(), OS);
> + for (const auto &Ref : Refs) {
> + OS.write(static_cast<unsigned char>(Ref.Kind));
> + writeLocation(Ref.Location, Strings, OS);
> + }
> +}
> +
> +std::pair<SymbolID, std::vector<Ref>> readRefs(Reader &Data,
> + ArrayRef<StringRef>
> Strings) {
> + std::pair<SymbolID, std::vector<Ref>> Result;
> + Result.first = Data.consumeID();
> + Result.second.resize(Data.consumeVar());
> + for (auto &Ref : Result.second) {
> + Ref.Kind = static_cast<RefKind>(Data.consume8());
> + Ref.Location = readLocation(Data, Strings);
> + }
> + return Result;
> +}
> +
> // FILE ENCODING
> // A file is a RIFF chunk with type 'CdIx'.
> // It contains the sections:
> // - meta: version number
> // - stri: string table
> // - symb: symbols
> +// - refs: references to symbols
>
> // The current versioning scheme is simple - non-current versions are
> rejected.
> // If you make a breaking change, bump this version number to
> invalidate stored
> // data. Later we may want to support some backward compatibility.
> -constexpr static uint32_t Version = 4;
> +constexpr static uint32_t Version = 5;
>
> Expected<IndexFileIn> readRIFF(StringRef Data) {
> auto RIFF = riff::readFile(Data);
> @@ -342,6 +372,18 @@ Expected<IndexFileIn> readRIFF(StringRef
> return makeError("malformed or truncated symbol");
> Result.Symbols = std::move(Symbols).build();
> }
> + if (Chunks.count("refs")) {
> + Reader RefsReader(Chunks.lookup("refs"));
> + RefSlab::Builder Refs;
> + while (!RefsReader.eof()) {
> + auto RefsBundle = readRefs(RefsReader, Strings->Strings);
> + for (const auto &Ref : RefsBundle.second) // FIXME: bulk insert?
> + Refs.insert(RefsBundle.first, Ref);
> + }
> + if (RefsReader.err())
> + return makeError("malformed or truncated refs");
> + Result.Refs = std::move(Refs).build();
> + }
> return std::move(Result);
> }
>
> @@ -363,6 +405,14 @@ void writeRIFF(const IndexFileOut &Data,
> Symbols.emplace_back(Sym);
> visitStrings(Symbols.back(), [&](StringRef &S) {
> Strings.intern(S); });
> }
> + std::vector<std::pair<SymbolID, std::vector<Ref>>> Refs;
> + if (Data.Refs) {
> + for (const auto &Sym : *Data.Refs) {
> + Refs.emplace_back(Sym);
> + for (auto &Ref : Refs.back().second)
> + Strings.intern(Ref.Location.FileURI);
> + }
> + }
>
> std::string StringSection;
> {
> @@ -379,6 +429,16 @@ void writeRIFF(const IndexFileOut &Data,
> }
> RIFF.Chunks.push_back({riff::fourCC("symb"), SymbolSection});
>
> + std::string RefsSection;
> + if (Data.Refs) {
> + {
> + raw_string_ostream RefsOS(RefsSection);
> + for (const auto &Sym : Refs)
> + writeRefs(Sym.first, Sym.second, Strings, RefsOS);
> + }
> + RIFF.Chunks.push_back({riff::fourCC("refs"), RefsSection});
> + }
> +
> OS << RIFF;
> }
>
> @@ -428,6 +488,8 @@ std::unique_ptr<SymbolIndex> loadIndex(l
> if (auto I = readIndexFile(Buffer->get()->getBuffer())) {
> if (I->Symbols)
> Symbols = std::move(*I->Symbols);
> + if (I->Refs)
> + Refs = std::move(*I->Refs);
> } else {
> llvm::errs() << "Bad Index: " << llvm::toString(I.takeError())
> << "\n";
> return nullptr;
>
> Modified: clang-tools-extra/trunk/clangd/index/Serialization.h
> URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> extra/trunk/clangd/index/Serialization.h?rev=343778&r1=343777&r2=343778
> &view=diff
> =======================================================================
> =======
> --- clang-tools-extra/trunk/clangd/index/Serialization.h (original)
> +++ clang-tools-extra/trunk/clangd/index/Serialization.h Thu Oct 4
> 07:09:55 2018
> @@ -38,26 +38,29 @@ enum class IndexFileFormat {
> // Holds the contents of an index file that was read.
> struct IndexFileIn {
> llvm::Optional<SymbolSlab> Symbols;
> + llvm::Optional<RefSlab> Refs;
> };
> -// Parse an index file. The input must be a RIFF container chunk.
> +// Parse an index file. The input must be a RIFF or YAML file.
> llvm::Expected<IndexFileIn> readIndexFile(llvm::StringRef);
>
> // Specifies the contents of an index file to be written.
> struct IndexFileOut {
> - const SymbolSlab *Symbols;
> - // TODO: Support serializing symbol occurrences.
> + const SymbolSlab *Symbols = nullptr;
> + const RefSlab *Refs = nullptr;
> // TODO: Support serializing Dex posting lists.
> IndexFileFormat Format = IndexFileFormat::RIFF;
>
> IndexFileOut() = default;
> IndexFileOut(const IndexFileIn &I)
> - : Symbols(I.Symbols ? I.Symbols.getPointer() : nullptr) {}
> + : Symbols(I.Symbols ? I.Symbols.getPointer() : nullptr),
> + Refs(I.Refs ? I.Refs.getPointer() : nullptr) {}
> };
> // Serializes an index file.
> llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, const
> IndexFileOut &O);
>
> // Convert a single symbol to YAML, a nice debug representation.
> std::string toYAML(const Symbol &);
> +std::string toYAML(const std::pair<SymbolID, ArrayRef<Ref>> &);
>
> // Build an in-memory static index from an index file.
> // The size should be relatively small, so data can be managed in
> memory.
>
> Modified: clang-tools-extra/trunk/clangd/index/YAMLSerialization.cpp
> URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> extra/trunk/clangd/index/YAMLSerialization.cpp?rev=343778&r1=343777&r2=
> 343778&view=diff
> =======================================================================
> =======
> --- clang-tools-extra/trunk/clangd/index/YAMLSerialization.cpp
> (original)
> +++ clang-tools-extra/trunk/clangd/index/YAMLSerialization.cpp Thu Oct
> 4 07:09:55 2018
> @@ -6,6 +6,12 @@
> // License. See LICENSE.TXT for details.
> //
> //===-----------------------------------------------------------------
> -----===//
> +//
> +// A YAML index file is a sequence of tagged entries.
> +// Each entry either encodes a Symbol or the list of references to a
> symbol
> +// (a "ref bundle").
> +//
> +//===-----------------------------------------------------------------
> -----===//
>
> #include "Index.h"
> #include "Serialization.h"
> @@ -20,10 +26,22 @@
> #include <cstdint>
>
>
> LLVM_YAML_IS_SEQUENCE_VECTOR(clang::clangd::Symbol::IncludeHeaderWithRe
> ferences)
> +LLVM_YAML_IS_SEQUENCE_VECTOR(clang::clangd::Ref)
>
> +namespace {
> +using RefBundle =
> + std::pair<clang::clangd::SymbolID,
> std::vector<clang::clangd::Ref>>;
> +// This is a pale imitation of std::variant<Symbol, RefBundle>
> +struct VariantEntry {
> + llvm::Optional<clang::clangd::Symbol> Symbol;
> + llvm::Optional<RefBundle> Refs;
> +};
> +} // namespace
> namespace llvm {
> namespace yaml {
>
> +using clang::clangd::Ref;
> +using clang::clangd::RefKind;
> using clang::clangd::Symbol;
> using clang::clangd::SymbolID;
> using clang::clangd::SymbolLocation;
> @@ -179,6 +197,46 @@ template <> struct ScalarEnumerationTrai
> }
> };
>
> +template <> struct MappingTraits<RefBundle> {
> + static void mapping(IO &IO, RefBundle &Refs) {
> + MappingNormalization<NormalizedSymbolID, SymbolID> NSymbolID(IO,
> +
> Refs.first);
> + IO.mapRequired("ID", NSymbolID->HexString);
> + IO.mapRequired("References", Refs.second);
> + }
> +};
> +
> +struct NormalizedRefKind {
> + NormalizedRefKind(IO &) {}
> + NormalizedRefKind(IO &, RefKind O) { Kind = static_cast<uint8_t>(O);
> }
> +
> + RefKind denormalize(IO &) { return static_cast<RefKind>(Kind); }
> +
> + uint8_t Kind = 0;
> +};
> +
> +template <> struct MappingTraits<Ref> {
> + static void mapping(IO &IO, Ref &R) {
> + MappingNormalization<NormalizedRefKind, RefKind> NKind(IO,
> R.Kind);
> + IO.mapRequired("Kind", NKind->Kind);
> + IO.mapRequired("Location", R.Location);
> + }
> +};
> +
> +template <> struct MappingTraits<VariantEntry> {
> + static void mapping(IO &IO, VariantEntry &Variant) {
> + if (IO.mapTag("!Symbol", Variant.Symbol.hasValue())) {
> + if (!IO.outputting())
> + Variant.Symbol.emplace();
> + MappingTraits<Symbol>::mapping(IO, *Variant.Symbol);
> + } else if (IO.mapTag("!Refs", Variant.Refs.hasValue())) {
> + if (!IO.outputting())
> + Variant.Refs.emplace();
> + MappingTraits<RefBundle>::mapping(IO, *Variant.Refs);
> + }
> + }
> +};
> +
> } // namespace yaml
> } // namespace llvm
>
> @@ -187,23 +245,38 @@ namespace clangd {
>
> void writeYAML(const IndexFileOut &O, raw_ostream &OS) {
> llvm::yaml::Output Yout(OS);
> - for (Symbol Sym : *O.Symbols) // copy: Yout<< requires mutability.
> - Yout << Sym;
> + for (const auto &Sym : *O.Symbols) {
> + VariantEntry Entry;
> + Entry.Symbol = Sym;
> + Yout << Entry;
> + }
> + if (O.Refs)
> + for (auto &Sym : *O.Refs) {
> + VariantEntry Entry;
> + Entry.Refs = Sym;
> + Yout << Entry;
> + }
> }
>
> Expected<IndexFileIn> readYAML(StringRef Data) {
> SymbolSlab::Builder Symbols;
> + RefSlab::Builder Refs;
> llvm::yaml::Input Yin(Data);
> do {
> - Symbol S;
> - Yin >> S;
> + VariantEntry Variant;
> + Yin >> Variant;
> if (Yin.error())
> return llvm::errorCodeToError(Yin.error());
> - Symbols.insert(S);
> + if (Variant.Symbol)
> + Symbols.insert(*Variant.Symbol);
> + if (Variant.Refs)
> + for (const auto &Ref : Variant.Refs->second)
> + Refs.insert(Variant.Refs->first, Ref);
> } while (Yin.nextDocument());
>
> IndexFileIn Result;
> Result.Symbols.emplace(std::move(Symbols).build());
> + Result.Refs.emplace(std::move(Refs).build());
> return std::move(Result);
> }
>
> @@ -217,6 +290,17 @@ std::string toYAML(const Symbol &S) {
> }
> return Buf;
> }
> +
> +std::string toYAML(const std::pair<SymbolID, ArrayRef<Ref>> &Data) {
> + RefBundle Refs = {Data.first, Data.second};
> + std::string Buf;
> + {
> + llvm::raw_string_ostream OS(Buf);
> + llvm::yaml::Output Yout(OS);
> + Yout << Refs;
> + }
> + return Buf;
> +}
>
> } // namespace clangd
> } // namespace clang
>
> Modified: clang-tools-extra/trunk/clangd/indexer/IndexerMain.cpp
> URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> extra/trunk/clangd/indexer/IndexerMain.cpp?rev=343778&r1=343777&r2=3437
> 78&view=diff
> =======================================================================
> =======
> --- clang-tools-extra/trunk/clangd/indexer/IndexerMain.cpp (original)
> +++ clang-tools-extra/trunk/clangd/indexer/IndexerMain.cpp Thu Oct 4
> 07:09:55 2018
> @@ -67,18 +67,30 @@ public:
> else
> Symbols.insert(Sym);
> }
> + },
> + [&](RefSlab S) {
> + std::lock_guard<std::mutex> Lock(SymbolsMu);
> + for (const auto &Sym : S) {
> + // No need to merge as currently all Refs are from
> main file.
> + for (const auto &Ref : Sym.second)
> + Refs.insert(Sym.first, Ref);
> + }
> })
> .release();
> }
>
> // Awkward: we write the result in the destructor, because the
> executor
> // takes ownership so it's the easiest way to get our data back out.
> - ~IndexActionFactory() { Result.Symbols = std::move(Symbols).build();
> }
> + ~IndexActionFactory() {
> + Result.Symbols = std::move(Symbols).build();
> + Result.Refs = std::move(Refs).build();
> + }
>
> private:
> IndexFileIn &Result;
> std::mutex SymbolsMu;
> SymbolSlab::Builder Symbols;
> + RefSlab::Builder Refs;
> };
>
> } // namespace
>
> Modified: clang-tools-
> extra/trunk/unittests/clangd/SerializationTests.cpp
> URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> extra/trunk/unittests/clangd/SerializationTests.cpp?rev=343778&r1=34377
> 7&r2=343778&view=diff
> =======================================================================
> =======
> --- clang-tools-extra/trunk/unittests/clangd/SerializationTests.cpp
> (original)
> +++ clang-tools-extra/trunk/unittests/clangd/SerializationTests.cpp Thu
> Oct 4 07:09:55 2018
> @@ -13,6 +13,9 @@
> #include "gmock/gmock.h"
> #include "gtest/gtest.h"
>
> +using testing::_;
> +using testing::AllOf;
> +using testing::Pair;
> using testing::UnorderedElementsAre;
> using testing::UnorderedElementsAreArray;
> namespace clang {
> @@ -21,6 +24,7 @@ namespace {
>
> const char *YAML = R"(
> ---
> +!Symbol
> ID: 057557CEBF6E6B2DD437FBF60CC58F352D1DF856
> Name: 'Foo1'
> Scope: 'clang::'
> @@ -46,6 +50,7 @@ IncludeHeaders:
> References: 3
> ...
> ---
> +!Symbol
> ID: 057557CEBF6E6B2DD437FBF60CC58F352D1DF858
> Name: 'Foo2'
> Scope: 'clang::'
> @@ -64,6 +69,18 @@ Flags: 2
> Signature: '-sig'
> CompletionSnippetSuffix: '-snippet'
> ...
> +!Refs
> +ID: 057557CEBF6E6B2DD437FBF60CC58F352D1DF856
> +References:
> + - Kind: 4
> + Location:
> + FileURI: file:///path/foo.cc
> + Start:
> + Line: 5
> + Column: 3
> + End:
> + Line: 5
> + Column: 8
> )";
>
> MATCHER_P(ID, I, "") { return arg.ID ==
> cantFail(SymbolID::fromStr(I)); }
> @@ -107,6 +124,16 @@ TEST(SerializationTest, YAMLConversions)
> EXPECT_EQ(Sym2.CanonicalDeclaration.FileURI, "file:///path/bar.h");
> EXPECT_FALSE(Sym2.Flags & Symbol::IndexedForCodeCompletion);
> EXPECT_TRUE(Sym2.Flags & Symbol::Deprecated);
> +
> + ASSERT_TRUE(bool(ParsedYAML->Refs));
> + EXPECT_THAT(*ParsedYAML->Refs,
> + UnorderedElementsAre(
> + Pair(cantFail(SymbolID::fromStr(
> +
> "057557CEBF6E6B2DD437FBF60CC58F352D1DF856")),
> + testing::SizeIs(1))));
> + auto Ref1 = ParsedYAML->Refs->begin()->second.front();
> + EXPECT_EQ(Ref1.Kind, RefKind::Reference);
> + EXPECT_EQ(Ref1.Location.FileURI, "file:///path/foo.cc");
> }
>
> std::vector<std::string> YAMLFromSymbols(const SymbolSlab &Slab) {
> @@ -115,24 +142,37 @@ std::vector<std::string> YAMLFromSymbols
> Result.push_back(toYAML(Sym));
> return Result;
> }
> +std::vector<std::string> YAMLFromRefs(const RefSlab &Slab) {
> + std::vector<std::string> Result;
> + for (const auto &Sym : Slab)
> + Result.push_back(toYAML(Sym));
> + return Result;
> +}
>
> TEST(SerializationTest, BinaryConversions) {
> auto In = readIndexFile(YAML);
> EXPECT_TRUE(bool(In)) << In.takeError();
>
> // Write to binary format, and parse again.
> - IndexFileOut Out;
> - Out.Symbols = In->Symbols.getPointer();
> + IndexFileOut Out(*In);
> Out.Format = IndexFileFormat::RIFF;
> std::string Serialized = llvm::to_string(Out);
> + {
> + std::error_code EC;
> + llvm::raw_fd_ostream F("/tmp/foo", EC);
> + F << Serialized;
> + }
>
> auto In2 = readIndexFile(Serialized);
> ASSERT_TRUE(bool(In2)) << In.takeError();
> - ASSERT_TRUE(In->Symbols);
> + ASSERT_TRUE(In2->Symbols);
> + ASSERT_TRUE(In2->Refs);
>
> // Assert the YAML serializations match, for nice comparisons and
> diffs.
> EXPECT_THAT(YAMLFromSymbols(*In2->Symbols),
> UnorderedElementsAreArray(YAMLFromSymbols(*In-
> >Symbols)));
> + EXPECT_THAT(YAMLFromRefs(*In2->Refs),
> + UnorderedElementsAreArray(YAMLFromRefs(*In->Refs)));
> }
>
> } // namespace
>
>
> _______________________________________________
> cfe-commits mailing list
> cfe-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
More information about the cfe-commits
mailing list