[clang-tools-extra] r343778 - [clangd] clangd-indexer gathers refs and stores them in index files.
Sam McCall via cfe-commits
cfe-commits at lists.llvm.org
Fri Oct 5 05:24:28 PDT 2018
Thanks! There was actually some old debugging code that got left over in a
test. Sorry about that!
Fixed in r343845.
On Fri, Oct 5, 2018 at 11:34 AM <douglas.yung at sony.com> wrote:
> Hi Sam,
>
> Our internal build bot hit an assertion failure in the changes you made in
> this commit that I was able to reproduce by building your change on my
> machine on Windows 7 using Visual Studio 2015. None of the public build
> bots seem to have hit this issue, though I am not sure why at the moment.
>
> Here is the assertion failure that is hit while running ClangdTests.exe:
>
> [----------] 2 tests from SerializationTest
> [ RUN ] SerializationTest.YAMLConversions
> [ OK ] SerializationTest.YAMLConversions (0 ms)
> [ RUN ] SerializationTest.BinaryConversions
> Assertion failed: OutBufCur == OutBufStart && "raw_ostream destructor
> called with non-empty buffer!", file
> C:\src\upstream\llvm2\lib\Support\raw_ostream.cpp, line 73
> 0x000000013FA74405 (0x0000000000000016 0x0000000000000200
> 0x000006D100230006 0x000007FEEA9BAA51), HandleAbort() + 0x5 bytes(s),
> c:\src\upstream\llvm2\lib\support\windows\signals.inc, line 409
> 0x000007FEEAA1DC17 (0x0000000000000001 0x0000000100000000
> 0x0000000000000000 0x00000000009BF460), raise() + 0x1E7 bytes(s)
> 0x000007FEEAA1EAA1 (0x000007FE00000003 0x000007FE00000003
> 0x00000001413AF810 0x00000001413AF790), abort() + 0x31 bytes(s)
> 0x000007FEEAA20751 (0x0000000000000049 0x00000001413AF810
> 0x0000000000000122 0x000007FEEA9C05D6), _get_wpgmptr() + 0x1BE1 bytes(s)
> 0x000007FEEAA20A5F (0x00000000009BF890 0x00000000009BF680
> 0x0000000000B05AB0 0x000000014126F8C5), _wassert() + 0x3F bytes(s)
> 0x000000013FA40677 (0x0000000000000000 0x0000000000A80000
> 0x00000000009BF890 0x000000013FA41A2E), llvm::raw_ostream::~raw_ostream() +
> 0x37 bytes(s), c:\src\upstream\llvm2\lib\support\raw_ostream.cpp, line 75
> 0x000000013FA4057C (0x0000000004BA1FF0 0x00000000009BF680
> 0x0000000000B05AB0 0x0000000000B05AB0),
> llvm::raw_fd_ostream::~raw_fd_ostream() + 0x11C bytes(s),
> c:\src\upstream\llvm2\lib\support\raw_ostream.cpp, line 617
> 0x000000013F96A4CC (0x0000000004BA1FF0 0x0000000000B05AB0
> 0x0000000000B05AB0 0x0000000004BA1FF0), clang::clangd::`anonymous
> namespace'::SerializationTest_BinaryConversions_Test::TestBody() + 0x34C
> bytes(s),
> c:\src\upstream\llvm2\tools\clang\tools\extra\unittests\clangd\serializationtests.cpp,
> line 166
> 0x000000013FAE201C (0x0000000004BA1FF0 0x0000000000B05AB0
> 0x00000001413C4B68 0x000007FEEAA954B8),
> testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test,void>()
> + 0xC bytes(s),
> c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 2387 +
> 0x2 byte(s)
> 0x000000013FB09E5B (0x0000000004BA1FF0 0x0000000000B05AB0
> 0x00000001413C4D30 0x00000000009BF9E8), testing::Test::Run() + 0x7B
> bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc,
> line 2481
> 0x000000013FB0A0DB (0x0000016641B94CAF 0x0000000000B05AB0
> 0x0000000000B1DE30 0x0000000000AFF560), testing::TestInfo::Run() + 0xAB
> bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc,
> line 2660
> 0x000000013FB09F72 (0x0000000000000023 0x0000000000000000
> 0x0000000000B05AB0 0x0000000000AFF560), testing::TestCase::Run() + 0xB2
> bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc,
> line 2774 + 0x12 byte(s)
> 0x000000013FB0A529 (0x0000000000AFF410 0x0000000000000000
> 0x0000000000000001 0x000007FE00000001),
> testing::internal::UnitTestImpl::RunAllTests() + 0x229 bytes(s),
> c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 4649 +
> 0x39 byte(s)
> 0x000000013FAE213C (0x0000000000AFF410 0x0000000000000000
> 0x00000001413C54A8 0x0000000000000000),
> testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool>()
> + 0xC bytes(s),
> c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 2387 +
> 0x2 byte(s)
> 0x000000013FB0A27C (0x0000000000008002 0x0000000000000000
> 0x0000000000000000 0x0000000000000000), testing::UnitTest::Run() + 0xEC
> bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc,
> line 4257 + 0x17 byte(s)
> 0x0000000141299397 (0x0000000100000001 0x0000000000000000
> 0x0000000000000000 0x01D45C4625BC735E), main() + 0xB7 bytes(s),
> c:\src\upstream\llvm2\utils\unittest\unittestmain\testmain.cpp, line 52
> 0x00000001412707C1 (0x0000000000000000 0x0000000000000000
> 0x0000000000000000 0x0000000000000000), __scrt_common_main_seh() + 0x11D
> bytes(s), f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl, line 253
> + 0x22 byte(s)
> 0x0000000076C159CD (0x0000000000000000 0x0000000000000000
> 0x0000000000000000 0x0000000000000000), BaseThreadInitThunk() + 0xD bytes(s)
> 0x0000000076E7385D (0x0000000000000000 0x0000000000000000
> 0x0000000000000000 0x0000000000000000), RtlUserThreadStart() + 0x1D bytes(s)
>
> Could you take a look into this?
>
> Douglas Yung
>
> > -----Original Message-----
> > From: cfe-commits [mailto:cfe-commits-bounces at lists.llvm.org] On Behalf
> > Of Sam McCall via cfe-commits
> > Sent: Thursday, October 04, 2018 7:10
> > To: cfe-commits at lists.llvm.org
> > Subject: [clang-tools-extra] r343778 - [clangd] clangd-indexer gathers
> > refs and stores them in index files.
> >
> > Author: sammccall
> > Date: Thu Oct 4 07:09:55 2018
> > New Revision: 343778
> >
> > URL: http://llvm.org/viewvc/llvm-project?rev=343778&view=rev
> > Log:
> > [clangd] clangd-indexer gathers refs and stores them in index files.
> >
> > Reviewers: ioeric
> >
> > Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-
> > commits
> >
> > Differential Revision: https://reviews.llvm.org/D52531
> >
> > Modified:
> > clang-tools-extra/trunk/clangd/index/IndexAction.cpp
> > clang-tools-extra/trunk/clangd/index/IndexAction.h
> > clang-tools-extra/trunk/clangd/index/Serialization.cpp
> > clang-tools-extra/trunk/clangd/index/Serialization.h
> > clang-tools-extra/trunk/clangd/index/YAMLSerialization.cpp
> > clang-tools-extra/trunk/clangd/indexer/IndexerMain.cpp
> > clang-tools-extra/trunk/unittests/clangd/SerializationTests.cpp
> >
> > Modified: clang-tools-extra/trunk/clangd/index/IndexAction.cpp
> > URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> > extra/trunk/clangd/index/IndexAction.cpp?rev=343778&r1=343777&r2=343778
> > &view=diff
> > =======================================================================
> > =======
> > --- clang-tools-extra/trunk/clangd/index/IndexAction.cpp (original)
> > +++ clang-tools-extra/trunk/clangd/index/IndexAction.cpp Thu Oct 4
> > 07:09:55 2018
> > @@ -13,10 +13,11 @@ public:
> > IndexAction(std::shared_ptr<SymbolCollector> C,
> > std::unique_ptr<CanonicalIncludes> Includes,
> > const index::IndexingOptions &Opts,
> > - std::function<void(SymbolSlab)> &SymbolsCallback)
> > + std::function<void(SymbolSlab)> SymbolsCallback,
> > + std::function<void(RefSlab)> RefsCallback)
> > : WrapperFrontendAction(index::createIndexingAction(C, Opts,
> > nullptr)),
> > - SymbolsCallback(SymbolsCallback), Collector(C),
> > - Includes(std::move(Includes)),
> > + SymbolsCallback(SymbolsCallback), RefsCallback(RefsCallback),
> > + Collector(C), Includes(std::move(Includes)),
> > PragmaHandler(collectIWYUHeaderMaps(this->Includes.get())) {}
> >
> > std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
> > @@ -41,10 +42,13 @@ public:
> > return;
> > }
> > SymbolsCallback(Collector->takeSymbols());
> > + if (RefsCallback != nullptr)
> > + RefsCallback(Collector->takeRefs());
> > }
> >
> > private:
> > std::function<void(SymbolSlab)> SymbolsCallback;
> > + std::function<void(RefSlab)> RefsCallback;
> > std::shared_ptr<SymbolCollector> Collector;
> > std::unique_ptr<CanonicalIncludes> Includes;
> > std::unique_ptr<CommentHandler> PragmaHandler;
> > @@ -54,20 +58,23 @@ private:
> >
> > std::unique_ptr<FrontendAction>
> > createStaticIndexingAction(SymbolCollector::Options Opts,
> > - std::function<void(SymbolSlab)>
> > SymbolsCallback) {
> > + std::function<void(SymbolSlab)>
> > SymbolsCallback,
> > + std::function<void(RefSlab)> RefsCallback)
> > {
> > index::IndexingOptions IndexOpts;
> > IndexOpts.SystemSymbolFilter =
> > index::IndexingOptions::SystemSymbolFilterKind::All;
> > Opts.CollectIncludePath = true;
> > Opts.CountReferences = true;
> > Opts.Origin = SymbolOrigin::Static;
> > + if (RefsCallback != nullptr)
> > + Opts.RefFilter = RefKind::All;
> > auto Includes = llvm::make_unique<CanonicalIncludes>();
> > addSystemHeadersMapping(Includes.get());
> > Opts.Includes = Includes.get();
> > return llvm::make_unique<IndexAction>(
> > std::make_shared<SymbolCollector>(std::move(Opts)),
> > std::move(Includes),
> > - IndexOpts, SymbolsCallback);
> > -}
> > + IndexOpts, SymbolsCallback, RefsCallback);
> > +};
> >
> > } // namespace clangd
> > } // namespace clang
> >
> > Modified: clang-tools-extra/trunk/clangd/index/IndexAction.h
> > URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> > extra/trunk/clangd/index/IndexAction.h?rev=343778&r1=343777&r2=343778&v
> > iew=diff
> > =======================================================================
> > =======
> > --- clang-tools-extra/trunk/clangd/index/IndexAction.h (original)
> > +++ clang-tools-extra/trunk/clangd/index/IndexAction.h Thu Oct 4
> > 07:09:55 2018
> > @@ -21,10 +21,13 @@ namespace clangd {
> > // Only a subset of SymbolCollector::Options are respected:
> > // - include paths are always collected, and canonicalized
> > appropriately
> > // - references are always counted
> > +// - main-file refs are collected (if RefsCallback is non-null)
> > // - the symbol origin is always Static
> > +// FIXME: refs from headers should also be collected.
> > std::unique_ptr<FrontendAction>
> > createStaticIndexingAction(SymbolCollector::Options Opts,
> > - std::function<void(SymbolSlab)>
> > SymbolsCallback);
> > + std::function<void(SymbolSlab)>
> > SymbolsCallback,
> > + std::function<void(RefSlab)> RefsCallback);
> >
> > } // namespace clangd
> > } // namespace clang
> >
> > Modified: clang-tools-extra/trunk/clangd/index/Serialization.cpp
> > URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> > extra/trunk/clangd/index/Serialization.cpp?rev=343778&r1=343777&r2=3437
> > 78&view=diff
> > =======================================================================
> > =======
> > --- clang-tools-extra/trunk/clangd/index/Serialization.cpp (original)
> > +++ clang-tools-extra/trunk/clangd/index/Serialization.cpp Thu Oct 4
> > 07:09:55 2018
> > @@ -298,17 +298,47 @@ Symbol readSymbol(Reader &Data, ArrayRef
> > return Sym;
> > }
> >
> > +// REFS ENCODING
> > +// A refs section has data grouped by Symbol. Each symbol has:
> > +// - SymbolID: 20 bytes
> > +// - NumRefs: varint
> > +// - Ref[NumRefs]
> > +// Fields of Ref are encoded in turn, see implementation.
> > +
> > +void writeRefs(const SymbolID &ID, ArrayRef<Ref> Refs,
> > + const StringTableOut &Strings, raw_ostream &OS) {
> > + OS << ID.raw();
> > + writeVar(Refs.size(), OS);
> > + for (const auto &Ref : Refs) {
> > + OS.write(static_cast<unsigned char>(Ref.Kind));
> > + writeLocation(Ref.Location, Strings, OS);
> > + }
> > +}
> > +
> > +std::pair<SymbolID, std::vector<Ref>> readRefs(Reader &Data,
> > + ArrayRef<StringRef>
> > Strings) {
> > + std::pair<SymbolID, std::vector<Ref>> Result;
> > + Result.first = Data.consumeID();
> > + Result.second.resize(Data.consumeVar());
> > + for (auto &Ref : Result.second) {
> > + Ref.Kind = static_cast<RefKind>(Data.consume8());
> > + Ref.Location = readLocation(Data, Strings);
> > + }
> > + return Result;
> > +}
> > +
> > // FILE ENCODING
> > // A file is a RIFF chunk with type 'CdIx'.
> > // It contains the sections:
> > // - meta: version number
> > // - stri: string table
> > // - symb: symbols
> > +// - refs: references to symbols
> >
> > // The current versioning scheme is simple - non-current versions are
> > rejected.
> > // If you make a breaking change, bump this version number to
> > invalidate stored
> > // data. Later we may want to support some backward compatibility.
> > -constexpr static uint32_t Version = 4;
> > +constexpr static uint32_t Version = 5;
> >
> > Expected<IndexFileIn> readRIFF(StringRef Data) {
> > auto RIFF = riff::readFile(Data);
> > @@ -342,6 +372,18 @@ Expected<IndexFileIn> readRIFF(StringRef
> > return makeError("malformed or truncated symbol");
> > Result.Symbols = std::move(Symbols).build();
> > }
> > + if (Chunks.count("refs")) {
> > + Reader RefsReader(Chunks.lookup("refs"));
> > + RefSlab::Builder Refs;
> > + while (!RefsReader.eof()) {
> > + auto RefsBundle = readRefs(RefsReader, Strings->Strings);
> > + for (const auto &Ref : RefsBundle.second) // FIXME: bulk insert?
> > + Refs.insert(RefsBundle.first, Ref);
> > + }
> > + if (RefsReader.err())
> > + return makeError("malformed or truncated refs");
> > + Result.Refs = std::move(Refs).build();
> > + }
> > return std::move(Result);
> > }
> >
> > @@ -363,6 +405,14 @@ void writeRIFF(const IndexFileOut &Data,
> > Symbols.emplace_back(Sym);
> > visitStrings(Symbols.back(), [&](StringRef &S) {
> > Strings.intern(S); });
> > }
> > + std::vector<std::pair<SymbolID, std::vector<Ref>>> Refs;
> > + if (Data.Refs) {
> > + for (const auto &Sym : *Data.Refs) {
> > + Refs.emplace_back(Sym);
> > + for (auto &Ref : Refs.back().second)
> > + Strings.intern(Ref.Location.FileURI);
> > + }
> > + }
> >
> > std::string StringSection;
> > {
> > @@ -379,6 +429,16 @@ void writeRIFF(const IndexFileOut &Data,
> > }
> > RIFF.Chunks.push_back({riff::fourCC("symb"), SymbolSection});
> >
> > + std::string RefsSection;
> > + if (Data.Refs) {
> > + {
> > + raw_string_ostream RefsOS(RefsSection);
> > + for (const auto &Sym : Refs)
> > + writeRefs(Sym.first, Sym.second, Strings, RefsOS);
> > + }
> > + RIFF.Chunks.push_back({riff::fourCC("refs"), RefsSection});
> > + }
> > +
> > OS << RIFF;
> > }
> >
> > @@ -428,6 +488,8 @@ std::unique_ptr<SymbolIndex> loadIndex(l
> > if (auto I = readIndexFile(Buffer->get()->getBuffer())) {
> > if (I->Symbols)
> > Symbols = std::move(*I->Symbols);
> > + if (I->Refs)
> > + Refs = std::move(*I->Refs);
> > } else {
> > llvm::errs() << "Bad Index: " << llvm::toString(I.takeError())
> > << "\n";
> > return nullptr;
> >
> > Modified: clang-tools-extra/trunk/clangd/index/Serialization.h
> > URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> > extra/trunk/clangd/index/Serialization.h?rev=343778&r1=343777&r2=343778
> > &view=diff
> > =======================================================================
> > =======
> > --- clang-tools-extra/trunk/clangd/index/Serialization.h (original)
> > +++ clang-tools-extra/trunk/clangd/index/Serialization.h Thu Oct 4
> > 07:09:55 2018
> > @@ -38,26 +38,29 @@ enum class IndexFileFormat {
> > // Holds the contents of an index file that was read.
> > struct IndexFileIn {
> > llvm::Optional<SymbolSlab> Symbols;
> > + llvm::Optional<RefSlab> Refs;
> > };
> > -// Parse an index file. The input must be a RIFF container chunk.
> > +// Parse an index file. The input must be a RIFF or YAML file.
> > llvm::Expected<IndexFileIn> readIndexFile(llvm::StringRef);
> >
> > // Specifies the contents of an index file to be written.
> > struct IndexFileOut {
> > - const SymbolSlab *Symbols;
> > - // TODO: Support serializing symbol occurrences.
> > + const SymbolSlab *Symbols = nullptr;
> > + const RefSlab *Refs = nullptr;
> > // TODO: Support serializing Dex posting lists.
> > IndexFileFormat Format = IndexFileFormat::RIFF;
> >
> > IndexFileOut() = default;
> > IndexFileOut(const IndexFileIn &I)
> > - : Symbols(I.Symbols ? I.Symbols.getPointer() : nullptr) {}
> > + : Symbols(I.Symbols ? I.Symbols.getPointer() : nullptr),
> > + Refs(I.Refs ? I.Refs.getPointer() : nullptr) {}
> > };
> > // Serializes an index file.
> > llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, const
> > IndexFileOut &O);
> >
> > // Convert a single symbol to YAML, a nice debug representation.
> > std::string toYAML(const Symbol &);
> > +std::string toYAML(const std::pair<SymbolID, ArrayRef<Ref>> &);
> >
> > // Build an in-memory static index from an index file.
> > // The size should be relatively small, so data can be managed in
> > memory.
> >
> > Modified: clang-tools-extra/trunk/clangd/index/YAMLSerialization.cpp
> > URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> > extra/trunk/clangd/index/YAMLSerialization.cpp?rev=343778&r1=343777&r2=
> > 343778&view=diff
> > =======================================================================
> > =======
> > --- clang-tools-extra/trunk/clangd/index/YAMLSerialization.cpp
> > (original)
> > +++ clang-tools-extra/trunk/clangd/index/YAMLSerialization.cpp Thu Oct
> > 4 07:09:55 2018
> > @@ -6,6 +6,12 @@
> > // License. See LICENSE.TXT for details.
> > //
> > //===-----------------------------------------------------------------
> > -----===//
> > +//
> > +// A YAML index file is a sequence of tagged entries.
> > +// Each entry either encodes a Symbol or the list of references to a
> > symbol
> > +// (a "ref bundle").
> > +//
> > +//===-----------------------------------------------------------------
> > -----===//
> >
> > #include "Index.h"
> > #include "Serialization.h"
> > @@ -20,10 +26,22 @@
> > #include <cstdint>
> >
> >
> > LLVM_YAML_IS_SEQUENCE_VECTOR(clang::clangd::Symbol::IncludeHeaderWithRe
> > ferences)
> > +LLVM_YAML_IS_SEQUENCE_VECTOR(clang::clangd::Ref)
> >
> > +namespace {
> > +using RefBundle =
> > + std::pair<clang::clangd::SymbolID,
> > std::vector<clang::clangd::Ref>>;
> > +// This is a pale imitation of std::variant<Symbol, RefBundle>
> > +struct VariantEntry {
> > + llvm::Optional<clang::clangd::Symbol> Symbol;
> > + llvm::Optional<RefBundle> Refs;
> > +};
> > +} // namespace
> > namespace llvm {
> > namespace yaml {
> >
> > +using clang::clangd::Ref;
> > +using clang::clangd::RefKind;
> > using clang::clangd::Symbol;
> > using clang::clangd::SymbolID;
> > using clang::clangd::SymbolLocation;
> > @@ -179,6 +197,46 @@ template <> struct ScalarEnumerationTrai
> > }
> > };
> >
> > +template <> struct MappingTraits<RefBundle> {
> > + static void mapping(IO &IO, RefBundle &Refs) {
> > + MappingNormalization<NormalizedSymbolID, SymbolID> NSymbolID(IO,
> > +
> > Refs.first);
> > + IO.mapRequired("ID", NSymbolID->HexString);
> > + IO.mapRequired("References", Refs.second);
> > + }
> > +};
> > +
> > +struct NormalizedRefKind {
> > + NormalizedRefKind(IO &) {}
> > + NormalizedRefKind(IO &, RefKind O) { Kind = static_cast<uint8_t>(O);
> > }
> > +
> > + RefKind denormalize(IO &) { return static_cast<RefKind>(Kind); }
> > +
> > + uint8_t Kind = 0;
> > +};
> > +
> > +template <> struct MappingTraits<Ref> {
> > + static void mapping(IO &IO, Ref &R) {
> > + MappingNormalization<NormalizedRefKind, RefKind> NKind(IO,
> > R.Kind);
> > + IO.mapRequired("Kind", NKind->Kind);
> > + IO.mapRequired("Location", R.Location);
> > + }
> > +};
> > +
> > +template <> struct MappingTraits<VariantEntry> {
> > + static void mapping(IO &IO, VariantEntry &Variant) {
> > + if (IO.mapTag("!Symbol", Variant.Symbol.hasValue())) {
> > + if (!IO.outputting())
> > + Variant.Symbol.emplace();
> > + MappingTraits<Symbol>::mapping(IO, *Variant.Symbol);
> > + } else if (IO.mapTag("!Refs", Variant.Refs.hasValue())) {
> > + if (!IO.outputting())
> > + Variant.Refs.emplace();
> > + MappingTraits<RefBundle>::mapping(IO, *Variant.Refs);
> > + }
> > + }
> > +};
> > +
> > } // namespace yaml
> > } // namespace llvm
> >
> > @@ -187,23 +245,38 @@ namespace clangd {
> >
> > void writeYAML(const IndexFileOut &O, raw_ostream &OS) {
> > llvm::yaml::Output Yout(OS);
> > - for (Symbol Sym : *O.Symbols) // copy: Yout<< requires mutability.
> > - Yout << Sym;
> > + for (const auto &Sym : *O.Symbols) {
> > + VariantEntry Entry;
> > + Entry.Symbol = Sym;
> > + Yout << Entry;
> > + }
> > + if (O.Refs)
> > + for (auto &Sym : *O.Refs) {
> > + VariantEntry Entry;
> > + Entry.Refs = Sym;
> > + Yout << Entry;
> > + }
> > }
> >
> > Expected<IndexFileIn> readYAML(StringRef Data) {
> > SymbolSlab::Builder Symbols;
> > + RefSlab::Builder Refs;
> > llvm::yaml::Input Yin(Data);
> > do {
> > - Symbol S;
> > - Yin >> S;
> > + VariantEntry Variant;
> > + Yin >> Variant;
> > if (Yin.error())
> > return llvm::errorCodeToError(Yin.error());
> > - Symbols.insert(S);
> > + if (Variant.Symbol)
> > + Symbols.insert(*Variant.Symbol);
> > + if (Variant.Refs)
> > + for (const auto &Ref : Variant.Refs->second)
> > + Refs.insert(Variant.Refs->first, Ref);
> > } while (Yin.nextDocument());
> >
> > IndexFileIn Result;
> > Result.Symbols.emplace(std::move(Symbols).build());
> > + Result.Refs.emplace(std::move(Refs).build());
> > return std::move(Result);
> > }
> >
> > @@ -217,6 +290,17 @@ std::string toYAML(const Symbol &S) {
> > }
> > return Buf;
> > }
> > +
> > +std::string toYAML(const std::pair<SymbolID, ArrayRef<Ref>> &Data) {
> > + RefBundle Refs = {Data.first, Data.second};
> > + std::string Buf;
> > + {
> > + llvm::raw_string_ostream OS(Buf);
> > + llvm::yaml::Output Yout(OS);
> > + Yout << Refs;
> > + }
> > + return Buf;
> > +}
> >
> > } // namespace clangd
> > } // namespace clang
> >
> > Modified: clang-tools-extra/trunk/clangd/indexer/IndexerMain.cpp
> > URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> > extra/trunk/clangd/indexer/IndexerMain.cpp?rev=343778&r1=343777&r2=3437
> > 78&view=diff
> > =======================================================================
> > =======
> > --- clang-tools-extra/trunk/clangd/indexer/IndexerMain.cpp (original)
> > +++ clang-tools-extra/trunk/clangd/indexer/IndexerMain.cpp Thu Oct 4
> > 07:09:55 2018
> > @@ -67,18 +67,30 @@ public:
> > else
> > Symbols.insert(Sym);
> > }
> > + },
> > + [&](RefSlab S) {
> > + std::lock_guard<std::mutex> Lock(SymbolsMu);
> > + for (const auto &Sym : S) {
> > + // No need to merge as currently all Refs are from
> > main file.
> > + for (const auto &Ref : Sym.second)
> > + Refs.insert(Sym.first, Ref);
> > + }
> > })
> > .release();
> > }
> >
> > // Awkward: we write the result in the destructor, because the
> > executor
> > // takes ownership so it's the easiest way to get our data back out.
> > - ~IndexActionFactory() { Result.Symbols = std::move(Symbols).build();
> > }
> > + ~IndexActionFactory() {
> > + Result.Symbols = std::move(Symbols).build();
> > + Result.Refs = std::move(Refs).build();
> > + }
> >
> > private:
> > IndexFileIn &Result;
> > std::mutex SymbolsMu;
> > SymbolSlab::Builder Symbols;
> > + RefSlab::Builder Refs;
> > };
> >
> > } // namespace
> >
> > Modified: clang-tools-
> > extra/trunk/unittests/clangd/SerializationTests.cpp
> > URL: http://llvm.org/viewvc/llvm-project/clang-tools-
> > extra/trunk/unittests/clangd/SerializationTests.cpp?rev=343778&r1=34377
> > 7&r2=343778&view=diff
> > =======================================================================
> > =======
> > --- clang-tools-extra/trunk/unittests/clangd/SerializationTests.cpp
> > (original)
> > +++ clang-tools-extra/trunk/unittests/clangd/SerializationTests.cpp Thu
> > Oct 4 07:09:55 2018
> > @@ -13,6 +13,9 @@
> > #include "gmock/gmock.h"
> > #include "gtest/gtest.h"
> >
> > +using testing::_;
> > +using testing::AllOf;
> > +using testing::Pair;
> > using testing::UnorderedElementsAre;
> > using testing::UnorderedElementsAreArray;
> > namespace clang {
> > @@ -21,6 +24,7 @@ namespace {
> >
> > const char *YAML = R"(
> > ---
> > +!Symbol
> > ID: 057557CEBF6E6B2DD437FBF60CC58F352D1DF856
> > Name: 'Foo1'
> > Scope: 'clang::'
> > @@ -46,6 +50,7 @@ IncludeHeaders:
> > References: 3
> > ...
> > ---
> > +!Symbol
> > ID: 057557CEBF6E6B2DD437FBF60CC58F352D1DF858
> > Name: 'Foo2'
> > Scope: 'clang::'
> > @@ -64,6 +69,18 @@ Flags: 2
> > Signature: '-sig'
> > CompletionSnippetSuffix: '-snippet'
> > ...
> > +!Refs
> > +ID: 057557CEBF6E6B2DD437FBF60CC58F352D1DF856
> > +References:
> > + - Kind: 4
> > + Location:
> > + FileURI: file:///path/foo.cc
> > + Start:
> > + Line: 5
> > + Column: 3
> > + End:
> > + Line: 5
> > + Column: 8
> > )";
> >
> > MATCHER_P(ID, I, "") { return arg.ID ==
> > cantFail(SymbolID::fromStr(I)); }
> > @@ -107,6 +124,16 @@ TEST(SerializationTest, YAMLConversions)
> > EXPECT_EQ(Sym2.CanonicalDeclaration.FileURI, "file:///path/bar.h");
> > EXPECT_FALSE(Sym2.Flags & Symbol::IndexedForCodeCompletion);
> > EXPECT_TRUE(Sym2.Flags & Symbol::Deprecated);
> > +
> > + ASSERT_TRUE(bool(ParsedYAML->Refs));
> > + EXPECT_THAT(*ParsedYAML->Refs,
> > + UnorderedElementsAre(
> > + Pair(cantFail(SymbolID::fromStr(
> > +
> > "057557CEBF6E6B2DD437FBF60CC58F352D1DF856")),
> > + testing::SizeIs(1))));
> > + auto Ref1 = ParsedYAML->Refs->begin()->second.front();
> > + EXPECT_EQ(Ref1.Kind, RefKind::Reference);
> > + EXPECT_EQ(Ref1.Location.FileURI, "file:///path/foo.cc");
> > }
> >
> > std::vector<std::string> YAMLFromSymbols(const SymbolSlab &Slab) {
> > @@ -115,24 +142,37 @@ std::vector<std::string> YAMLFromSymbols
> > Result.push_back(toYAML(Sym));
> > return Result;
> > }
> > +std::vector<std::string> YAMLFromRefs(const RefSlab &Slab) {
> > + std::vector<std::string> Result;
> > + for (const auto &Sym : Slab)
> > + Result.push_back(toYAML(Sym));
> > + return Result;
> > +}
> >
> > TEST(SerializationTest, BinaryConversions) {
> > auto In = readIndexFile(YAML);
> > EXPECT_TRUE(bool(In)) << In.takeError();
> >
> > // Write to binary format, and parse again.
> > - IndexFileOut Out;
> > - Out.Symbols = In->Symbols.getPointer();
> > + IndexFileOut Out(*In);
> > Out.Format = IndexFileFormat::RIFF;
> > std::string Serialized = llvm::to_string(Out);
> > + {
> > + std::error_code EC;
> > + llvm::raw_fd_ostream F("/tmp/foo", EC);
> > + F << Serialized;
> > + }
> >
> > auto In2 = readIndexFile(Serialized);
> > ASSERT_TRUE(bool(In2)) << In.takeError();
> > - ASSERT_TRUE(In->Symbols);
> > + ASSERT_TRUE(In2->Symbols);
> > + ASSERT_TRUE(In2->Refs);
> >
> > // Assert the YAML serializations match, for nice comparisons and
> > diffs.
> > EXPECT_THAT(YAMLFromSymbols(*In2->Symbols),
> > UnorderedElementsAreArray(YAMLFromSymbols(*In-
> > >Symbols)));
> > + EXPECT_THAT(YAMLFromRefs(*In2->Refs),
> > + UnorderedElementsAreArray(YAMLFromRefs(*In->Refs)));
> > }
> >
> > } // namespace
> >
> >
> > _______________________________________________
> > cfe-commits mailing list
> > cfe-commits at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20181005/370a4ac3/attachment-0001.html>
More information about the cfe-commits
mailing list