<div dir="ltr"><div dir="ltr">Thanks! There was actually some old debugging code that got left over in a test. Sorry about that!<div>Fixed in r343845.</div></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Oct 5, 2018 at 11:34 AM <<a href="mailto:douglas.yung@sony.com">douglas.yung@sony.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Sam,<br>
<br>
Our internal build bot hit an assertion failure in the changes you made in this commit that I was able to reproduce by building your change on my machine on Windows 7 using Visual Studio 2015. None of the public build bots seem to have hit this issue, though I am not sure why at the moment.<br>
<br>
Here is the assertion failure that is hit while running ClangdTests.exe:<br>
<br>
[----------] 2 tests from SerializationTest<br>
[ RUN ] SerializationTest.YAMLConversions<br>
[ OK ] SerializationTest.YAMLConversions (0 ms)<br>
[ RUN ] SerializationTest.BinaryConversions<br>
Assertion failed: OutBufCur == OutBufStart && "raw_ostream destructor called with non-empty buffer!", file C:\src\upstream\llvm2\lib\Support\raw_ostream.cpp, line 73<br>
0x000000013FA74405 (0x0000000000000016 0x0000000000000200 0x000006D100230006 0x000007FEEA9BAA51), HandleAbort() + 0x5 bytes(s), c:\src\upstream\llvm2\lib\support\windows\signals.inc, line 409<br>
0x000007FEEAA1DC17 (0x0000000000000001 0x0000000100000000 0x0000000000000000 0x00000000009BF460), raise() + 0x1E7 bytes(s)<br>
0x000007FEEAA1EAA1 (0x000007FE00000003 0x000007FE00000003 0x00000001413AF810 0x00000001413AF790), abort() + 0x31 bytes(s)<br>
0x000007FEEAA20751 (0x0000000000000049 0x00000001413AF810 0x0000000000000122 0x000007FEEA9C05D6), _get_wpgmptr() + 0x1BE1 bytes(s)<br>
0x000007FEEAA20A5F (0x00000000009BF890 0x00000000009BF680 0x0000000000B05AB0 0x000000014126F8C5), _wassert() + 0x3F bytes(s)<br>
0x000000013FA40677 (0x0000000000000000 0x0000000000A80000 0x00000000009BF890 0x000000013FA41A2E), llvm::raw_ostream::~raw_ostream() + 0x37 bytes(s), c:\src\upstream\llvm2\lib\support\raw_ostream.cpp, line 75<br>
0x000000013FA4057C (0x0000000004BA1FF0 0x00000000009BF680 0x0000000000B05AB0 0x0000000000B05AB0), llvm::raw_fd_ostream::~raw_fd_ostream() + 0x11C bytes(s), c:\src\upstream\llvm2\lib\support\raw_ostream.cpp, line 617<br>
0x000000013F96A4CC (0x0000000004BA1FF0 0x0000000000B05AB0 0x0000000000B05AB0 0x0000000004BA1FF0), clang::clangd::`anonymous namespace'::SerializationTest_BinaryConversions_Test::TestBody() + 0x34C bytes(s), c:\src\upstream\llvm2\tools\clang\tools\extra\unittests\clangd\serializationtests.cpp, line 166<br>
0x000000013FAE201C (0x0000000004BA1FF0 0x0000000000B05AB0 0x00000001413C4B68 0x000007FEEAA954B8), testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test,void>() + 0xC bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 2387 + 0x2 byte(s)<br>
0x000000013FB09E5B (0x0000000004BA1FF0 0x0000000000B05AB0 0x00000001413C4D30 0x00000000009BF9E8), testing::Test::Run() + 0x7B bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 2481<br>
0x000000013FB0A0DB (0x0000016641B94CAF 0x0000000000B05AB0 0x0000000000B1DE30 0x0000000000AFF560), testing::TestInfo::Run() + 0xAB bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 2660<br>
0x000000013FB09F72 (0x0000000000000023 0x0000000000000000 0x0000000000B05AB0 0x0000000000AFF560), testing::TestCase::Run() + 0xB2 bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 2774 + 0x12 byte(s)<br>
0x000000013FB0A529 (0x0000000000AFF410 0x0000000000000000 0x0000000000000001 0x000007FE00000001), testing::internal::UnitTestImpl::RunAllTests() + 0x229 bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 4649 + 0x39 byte(s)<br>
0x000000013FAE213C (0x0000000000AFF410 0x0000000000000000 0x00000001413C54A8 0x0000000000000000), testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool>() + 0xC bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 2387 + 0x2 byte(s)<br>
0x000000013FB0A27C (0x0000000000008002 0x0000000000000000 0x0000000000000000 0x0000000000000000), testing::UnitTest::Run() + 0xEC bytes(s), c:\src\upstream\llvm2\utils\unittest\googletest\src\gtest.cc, line 4257 + 0x17 byte(s)<br>
0x0000000141299397 (0x0000000100000001 0x0000000000000000 0x0000000000000000 0x01D45C4625BC735E), main() + 0xB7 bytes(s), c:\src\upstream\llvm2\utils\unittest\unittestmain\testmain.cpp, line 52<br>
0x00000001412707C1 (0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000), __scrt_common_main_seh() + 0x11D bytes(s), f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl, line 253 + 0x22 byte(s)<br>
0x0000000076C159CD (0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000), BaseThreadInitThunk() + 0xD bytes(s)<br>
0x0000000076E7385D (0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000), RtlUserThreadStart() + 0x1D bytes(s)<br>
<br>
Could you take a look into this?<br>
<br>
Douglas Yung<br>
<br>
> -----Original Message-----<br>
> From: cfe-commits [mailto:<a href="mailto:cfe-commits-bounces@lists.llvm.org" target="_blank">cfe-commits-bounces@lists.llvm.org</a>] On Behalf<br>
> Of Sam McCall via cfe-commits<br>
> Sent: Thursday, October 04, 2018 7:10<br>
> To: <a href="mailto:cfe-commits@lists.llvm.org" target="_blank">cfe-commits@lists.llvm.org</a><br>
> Subject: [clang-tools-extra] r343778 - [clangd] clangd-indexer gathers<br>
> refs and stores them in index files.<br>
> <br>
> Author: sammccall<br>
> Date: Thu Oct 4 07:09:55 2018<br>
> New Revision: 343778<br>
> <br>
> URL: <a href="http://llvm.org/viewvc/llvm-project?rev=343778&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project?rev=343778&view=rev</a><br>
> Log:<br>
> [clangd] clangd-indexer gathers refs and stores them in index files.<br>
> <br>
> Reviewers: ioeric<br>
> <br>
> Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-<br>
> commits<br>
> <br>
> Differential Revision: <a href="https://reviews.llvm.org/D52531" rel="noreferrer" target="_blank">https://reviews.llvm.org/D52531</a><br>
> <br>
> Modified:<br>
> clang-tools-extra/trunk/clangd/index/IndexAction.cpp<br>
> clang-tools-extra/trunk/clangd/index/IndexAction.h<br>
> clang-tools-extra/trunk/clangd/index/Serialization.cpp<br>
> clang-tools-extra/trunk/clangd/index/Serialization.h<br>
> clang-tools-extra/trunk/clangd/index/YAMLSerialization.cpp<br>
> clang-tools-extra/trunk/clangd/indexer/IndexerMain.cpp<br>
> clang-tools-extra/trunk/unittests/clangd/SerializationTests.cpp<br>
> <br>
> Modified: clang-tools-extra/trunk/clangd/index/IndexAction.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/clang-tools-</a><br>
> extra/trunk/clangd/index/IndexAction.cpp?rev=343778&r1=343777&r2=343778<br>
> &view=diff<br>
> =======================================================================<br>
> =======<br>
> --- clang-tools-extra/trunk/clangd/index/IndexAction.cpp (original)<br>
> +++ clang-tools-extra/trunk/clangd/index/IndexAction.cpp Thu Oct 4<br>
> 07:09:55 2018<br>
> @@ -13,10 +13,11 @@ public:<br>
> IndexAction(std::shared_ptr<SymbolCollector> C,<br>
> std::unique_ptr<CanonicalIncludes> Includes,<br>
> const index::IndexingOptions &Opts,<br>
> - std::function<void(SymbolSlab)> &SymbolsCallback)<br>
> + std::function<void(SymbolSlab)> SymbolsCallback,<br>
> + std::function<void(RefSlab)> RefsCallback)<br>
> : WrapperFrontendAction(index::createIndexingAction(C, Opts,<br>
> nullptr)),<br>
> - SymbolsCallback(SymbolsCallback), Collector(C),<br>
> - Includes(std::move(Includes)),<br>
> + SymbolsCallback(SymbolsCallback), RefsCallback(RefsCallback),<br>
> + Collector(C), Includes(std::move(Includes)),<br>
> PragmaHandler(collectIWYUHeaderMaps(this->Includes.get())) {}<br>
> <br>
> std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,<br>
> @@ -41,10 +42,13 @@ public:<br>
> return;<br>
> }<br>
> SymbolsCallback(Collector->takeSymbols());<br>
> + if (RefsCallback != nullptr)<br>
> + RefsCallback(Collector->takeRefs());<br>
> }<br>
> <br>
> private:<br>
> std::function<void(SymbolSlab)> SymbolsCallback;<br>
> + std::function<void(RefSlab)> RefsCallback;<br>
> std::shared_ptr<SymbolCollector> Collector;<br>
> std::unique_ptr<CanonicalIncludes> Includes;<br>
> std::unique_ptr<CommentHandler> PragmaHandler;<br>
> @@ -54,20 +58,23 @@ private:<br>
> <br>
> std::unique_ptr<FrontendAction><br>
> createStaticIndexingAction(SymbolCollector::Options Opts,<br>
> - std::function<void(SymbolSlab)><br>
> SymbolsCallback) {<br>
> + std::function<void(SymbolSlab)><br>
> SymbolsCallback,<br>
> + std::function<void(RefSlab)> RefsCallback)<br>
> {<br>
> index::IndexingOptions IndexOpts;<br>
> IndexOpts.SystemSymbolFilter =<br>
> index::IndexingOptions::SystemSymbolFilterKind::All;<br>
> Opts.CollectIncludePath = true;<br>
> Opts.CountReferences = true;<br>
> Opts.Origin = SymbolOrigin::Static;<br>
> + if (RefsCallback != nullptr)<br>
> + Opts.RefFilter = RefKind::All;<br>
> auto Includes = llvm::make_unique<CanonicalIncludes>();<br>
> addSystemHeadersMapping(Includes.get());<br>
> Opts.Includes = Includes.get();<br>
> return llvm::make_unique<IndexAction>(<br>
> std::make_shared<SymbolCollector>(std::move(Opts)),<br>
> std::move(Includes),<br>
> - IndexOpts, SymbolsCallback);<br>
> -}<br>
> + IndexOpts, SymbolsCallback, RefsCallback);<br>
> +};<br>
> <br>
> } // namespace clangd<br>
> } // namespace clang<br>
> <br>
> Modified: clang-tools-extra/trunk/clangd/index/IndexAction.h<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/clang-tools-</a><br>
> extra/trunk/clangd/index/IndexAction.h?rev=343778&r1=343777&r2=343778&v<br>
> iew=diff<br>
> =======================================================================<br>
> =======<br>
> --- clang-tools-extra/trunk/clangd/index/IndexAction.h (original)<br>
> +++ clang-tools-extra/trunk/clangd/index/IndexAction.h Thu Oct 4<br>
> 07:09:55 2018<br>
> @@ -21,10 +21,13 @@ namespace clangd {<br>
> // Only a subset of SymbolCollector::Options are respected:<br>
> // - include paths are always collected, and canonicalized<br>
> appropriately<br>
> // - references are always counted<br>
> +// - main-file refs are collected (if RefsCallback is non-null)<br>
> // - the symbol origin is always Static<br>
> +// FIXME: refs from headers should also be collected.<br>
> std::unique_ptr<FrontendAction><br>
> createStaticIndexingAction(SymbolCollector::Options Opts,<br>
> - std::function<void(SymbolSlab)><br>
> SymbolsCallback);<br>
> + std::function<void(SymbolSlab)><br>
> SymbolsCallback,<br>
> + std::function<void(RefSlab)> RefsCallback);<br>
> <br>
> } // namespace clangd<br>
> } // namespace clang<br>
> <br>
> Modified: clang-tools-extra/trunk/clangd/index/Serialization.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/clang-tools-</a><br>
> extra/trunk/clangd/index/Serialization.cpp?rev=343778&r1=343777&r2=3437<br>
> 78&view=diff<br>
> =======================================================================<br>
> =======<br>
> --- clang-tools-extra/trunk/clangd/index/Serialization.cpp (original)<br>
> +++ clang-tools-extra/trunk/clangd/index/Serialization.cpp Thu Oct 4<br>
> 07:09:55 2018<br>
> @@ -298,17 +298,47 @@ Symbol readSymbol(Reader &Data, ArrayRef<br>
> return Sym;<br>
> }<br>
> <br>
> +// REFS ENCODING<br>
> +// A refs section has data grouped by Symbol. Each symbol has:<br>
> +// - SymbolID: 20 bytes<br>
> +// - NumRefs: varint<br>
> +// - Ref[NumRefs]<br>
> +// Fields of Ref are encoded in turn, see implementation.<br>
> +<br>
> +void writeRefs(const SymbolID &ID, ArrayRef<Ref> Refs,<br>
> + const StringTableOut &Strings, raw_ostream &OS) {<br>
> + OS << ID.raw();<br>
> + writeVar(Refs.size(), OS);<br>
> + for (const auto &Ref : Refs) {<br>
> + OS.write(static_cast<unsigned char>(Ref.Kind));<br>
> + writeLocation(Ref.Location, Strings, OS);<br>
> + }<br>
> +}<br>
> +<br>
> +std::pair<SymbolID, std::vector<Ref>> readRefs(Reader &Data,<br>
> + ArrayRef<StringRef><br>
> Strings) {<br>
> + std::pair<SymbolID, std::vector<Ref>> Result;<br>
> + Result.first = Data.consumeID();<br>
> + Result.second.resize(Data.consumeVar());<br>
> + for (auto &Ref : Result.second) {<br>
> + Ref.Kind = static_cast<RefKind>(Data.consume8());<br>
> + Ref.Location = readLocation(Data, Strings);<br>
> + }<br>
> + return Result;<br>
> +}<br>
> +<br>
> // FILE ENCODING<br>
> // A file is a RIFF chunk with type 'CdIx'.<br>
> // It contains the sections:<br>
> // - meta: version number<br>
> // - stri: string table<br>
> // - symb: symbols<br>
> +// - refs: references to symbols<br>
> <br>
> // The current versioning scheme is simple - non-current versions are<br>
> rejected.<br>
> // If you make a breaking change, bump this version number to<br>
> invalidate stored<br>
> // data. Later we may want to support some backward compatibility.<br>
> -constexpr static uint32_t Version = 4;<br>
> +constexpr static uint32_t Version = 5;<br>
> <br>
> Expected<IndexFileIn> readRIFF(StringRef Data) {<br>
> auto RIFF = riff::readFile(Data);<br>
> @@ -342,6 +372,18 @@ Expected<IndexFileIn> readRIFF(StringRef<br>
> return makeError("malformed or truncated symbol");<br>
> Result.Symbols = std::move(Symbols).build();<br>
> }<br>
> + if (Chunks.count("refs")) {<br>
> + Reader RefsReader(Chunks.lookup("refs"));<br>
> + RefSlab::Builder Refs;<br>
> + while (!RefsReader.eof()) {<br>
> + auto RefsBundle = readRefs(RefsReader, Strings->Strings);<br>
> + for (const auto &Ref : RefsBundle.second) // FIXME: bulk insert?<br>
> + Refs.insert(RefsBundle.first, Ref);<br>
> + }<br>
> + if (RefsReader.err())<br>
> + return makeError("malformed or truncated refs");<br>
> + Result.Refs = std::move(Refs).build();<br>
> + }<br>
> return std::move(Result);<br>
> }<br>
> <br>
> @@ -363,6 +405,14 @@ void writeRIFF(const IndexFileOut &Data,<br>
> Symbols.emplace_back(Sym);<br>
> visitStrings(Symbols.back(), [&](StringRef &S) {<br>
> Strings.intern(S); });<br>
> }<br>
> + std::vector<std::pair<SymbolID, std::vector<Ref>>> Refs;<br>
> + if (Data.Refs) {<br>
> + for (const auto &Sym : *Data.Refs) {<br>
> + Refs.emplace_back(Sym);<br>
> + for (auto &Ref : Refs.back().second)<br>
> + Strings.intern(Ref.Location.FileURI);<br>
> + }<br>
> + }<br>
> <br>
> std::string StringSection;<br>
> {<br>
> @@ -379,6 +429,16 @@ void writeRIFF(const IndexFileOut &Data,<br>
> }<br>
> RIFF.Chunks.push_back({riff::fourCC("symb"), SymbolSection});<br>
> <br>
> + std::string RefsSection;<br>
> + if (Data.Refs) {<br>
> + {<br>
> + raw_string_ostream RefsOS(RefsSection);<br>
> + for (const auto &Sym : Refs)<br>
> + writeRefs(Sym.first, Sym.second, Strings, RefsOS);<br>
> + }<br>
> + RIFF.Chunks.push_back({riff::fourCC("refs"), RefsSection});<br>
> + }<br>
> +<br>
> OS << RIFF;<br>
> }<br>
> <br>
> @@ -428,6 +488,8 @@ std::unique_ptr<SymbolIndex> loadIndex(l<br>
> if (auto I = readIndexFile(Buffer->get()->getBuffer())) {<br>
> if (I->Symbols)<br>
> Symbols = std::move(*I->Symbols);<br>
> + if (I->Refs)<br>
> + Refs = std::move(*I->Refs);<br>
> } else {<br>
> llvm::errs() << "Bad Index: " << llvm::toString(I.takeError())<br>
> << "\n";<br>
> return nullptr;<br>
> <br>
> Modified: clang-tools-extra/trunk/clangd/index/Serialization.h<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/clang-tools-</a><br>
> extra/trunk/clangd/index/Serialization.h?rev=343778&r1=343777&r2=343778<br>
> &view=diff<br>
> =======================================================================<br>
> =======<br>
> --- clang-tools-extra/trunk/clangd/index/Serialization.h (original)<br>
> +++ clang-tools-extra/trunk/clangd/index/Serialization.h Thu Oct 4<br>
> 07:09:55 2018<br>
> @@ -38,26 +38,29 @@ enum class IndexFileFormat {<br>
> // Holds the contents of an index file that was read.<br>
> struct IndexFileIn {<br>
> llvm::Optional<SymbolSlab> Symbols;<br>
> + llvm::Optional<RefSlab> Refs;<br>
> };<br>
> -// Parse an index file. The input must be a RIFF container chunk.<br>
> +// Parse an index file. The input must be a RIFF or YAML file.<br>
> llvm::Expected<IndexFileIn> readIndexFile(llvm::StringRef);<br>
> <br>
> // Specifies the contents of an index file to be written.<br>
> struct IndexFileOut {<br>
> - const SymbolSlab *Symbols;<br>
> - // TODO: Support serializing symbol occurrences.<br>
> + const SymbolSlab *Symbols = nullptr;<br>
> + const RefSlab *Refs = nullptr;<br>
> // TODO: Support serializing Dex posting lists.<br>
> IndexFileFormat Format = IndexFileFormat::RIFF;<br>
> <br>
> IndexFileOut() = default;<br>
> IndexFileOut(const IndexFileIn &I)<br>
> - : Symbols(I.Symbols ? I.Symbols.getPointer() : nullptr) {}<br>
> + : Symbols(I.Symbols ? I.Symbols.getPointer() : nullptr),<br>
> + Refs(I.Refs ? I.Refs.getPointer() : nullptr) {}<br>
> };<br>
> // Serializes an index file.<br>
> llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, const<br>
> IndexFileOut &O);<br>
> <br>
> // Convert a single symbol to YAML, a nice debug representation.<br>
> std::string toYAML(const Symbol &);<br>
> +std::string toYAML(const std::pair<SymbolID, ArrayRef<Ref>> &);<br>
> <br>
> // Build an in-memory static index from an index file.<br>
> // The size should be relatively small, so data can be managed in<br>
> memory.<br>
> <br>
> Modified: clang-tools-extra/trunk/clangd/index/YAMLSerialization.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/clang-tools-</a><br>
> extra/trunk/clangd/index/YAMLSerialization.cpp?rev=343778&r1=343777&r2=<br>
> 343778&view=diff<br>
> =======================================================================<br>
> =======<br>
> --- clang-tools-extra/trunk/clangd/index/YAMLSerialization.cpp<br>
> (original)<br>
> +++ clang-tools-extra/trunk/clangd/index/YAMLSerialization.cpp Thu Oct<br>
> 4 07:09:55 2018<br>
> @@ -6,6 +6,12 @@<br>
> // License. See LICENSE.TXT for details.<br>
> //<br>
> //===-----------------------------------------------------------------<br>
> -----===//<br>
> +//<br>
> +// A YAML index file is a sequence of tagged entries.<br>
> +// Each entry either encodes a Symbol or the list of references to a<br>
> symbol<br>
> +// (a "ref bundle").<br>
> +//<br>
> +//===-----------------------------------------------------------------<br>
> -----===//<br>
> <br>
> #include "Index.h"<br>
> #include "Serialization.h"<br>
> @@ -20,10 +26,22 @@<br>
> #include <cstdint><br>
> <br>
> <br>
> LLVM_YAML_IS_SEQUENCE_VECTOR(clang::clangd::Symbol::IncludeHeaderWithRe<br>
> ferences)<br>
> +LLVM_YAML_IS_SEQUENCE_VECTOR(clang::clangd::Ref)<br>
> <br>
> +namespace {<br>
> +using RefBundle =<br>
> + std::pair<clang::clangd::SymbolID,<br>
> std::vector<clang::clangd::Ref>>;<br>
> +// This is a pale imitation of std::variant<Symbol, RefBundle><br>
> +struct VariantEntry {<br>
> + llvm::Optional<clang::clangd::Symbol> Symbol;<br>
> + llvm::Optional<RefBundle> Refs;<br>
> +};<br>
> +} // namespace<br>
> namespace llvm {<br>
> namespace yaml {<br>
> <br>
> +using clang::clangd::Ref;<br>
> +using clang::clangd::RefKind;<br>
> using clang::clangd::Symbol;<br>
> using clang::clangd::SymbolID;<br>
> using clang::clangd::SymbolLocation;<br>
> @@ -179,6 +197,46 @@ template <> struct ScalarEnumerationTrai<br>
> }<br>
> };<br>
> <br>
> +template <> struct MappingTraits<RefBundle> {<br>
> + static void mapping(IO &IO, RefBundle &Refs) {<br>
> + MappingNormalization<NormalizedSymbolID, SymbolID> NSymbolID(IO,<br>
> +<br>
> Refs.first);<br>
> + IO.mapRequired("ID", NSymbolID->HexString);<br>
> + IO.mapRequired("References", Refs.second);<br>
> + }<br>
> +};<br>
> +<br>
> +struct NormalizedRefKind {<br>
> + NormalizedRefKind(IO &) {}<br>
> + NormalizedRefKind(IO &, RefKind O) { Kind = static_cast<uint8_t>(O);<br>
> }<br>
> +<br>
> + RefKind denormalize(IO &) { return static_cast<RefKind>(Kind); }<br>
> +<br>
> + uint8_t Kind = 0;<br>
> +};<br>
> +<br>
> +template <> struct MappingTraits<Ref> {<br>
> + static void mapping(IO &IO, Ref &R) {<br>
> + MappingNormalization<NormalizedRefKind, RefKind> NKind(IO,<br>
> R.Kind);<br>
> + IO.mapRequired("Kind", NKind->Kind);<br>
> + IO.mapRequired("Location", R.Location);<br>
> + }<br>
> +};<br>
> +<br>
> +template <> struct MappingTraits<VariantEntry> {<br>
> + static void mapping(IO &IO, VariantEntry &Variant) {<br>
> + if (IO.mapTag("!Symbol", Variant.Symbol.hasValue())) {<br>
> + if (!IO.outputting())<br>
> + Variant.Symbol.emplace();<br>
> + MappingTraits<Symbol>::mapping(IO, *Variant.Symbol);<br>
> + } else if (IO.mapTag("!Refs", Variant.Refs.hasValue())) {<br>
> + if (!IO.outputting())<br>
> + Variant.Refs.emplace();<br>
> + MappingTraits<RefBundle>::mapping(IO, *Variant.Refs);<br>
> + }<br>
> + }<br>
> +};<br>
> +<br>
> } // namespace yaml<br>
> } // namespace llvm<br>
> <br>
> @@ -187,23 +245,38 @@ namespace clangd {<br>
> <br>
> void writeYAML(const IndexFileOut &O, raw_ostream &OS) {<br>
> llvm::yaml::Output Yout(OS);<br>
> - for (Symbol Sym : *O.Symbols) // copy: Yout<< requires mutability.<br>
> - Yout << Sym;<br>
> + for (const auto &Sym : *O.Symbols) {<br>
> + VariantEntry Entry;<br>
> + Entry.Symbol = Sym;<br>
> + Yout << Entry;<br>
> + }<br>
> + if (O.Refs)<br>
> + for (auto &Sym : *O.Refs) {<br>
> + VariantEntry Entry;<br>
> + Entry.Refs = Sym;<br>
> + Yout << Entry;<br>
> + }<br>
> }<br>
> <br>
> Expected<IndexFileIn> readYAML(StringRef Data) {<br>
> SymbolSlab::Builder Symbols;<br>
> + RefSlab::Builder Refs;<br>
> llvm::yaml::Input Yin(Data);<br>
> do {<br>
> - Symbol S;<br>
> - Yin >> S;<br>
> + VariantEntry Variant;<br>
> + Yin >> Variant;<br>
> if (Yin.error())<br>
> return llvm::errorCodeToError(Yin.error());<br>
> - Symbols.insert(S);<br>
> + if (Variant.Symbol)<br>
> + Symbols.insert(*Variant.Symbol);<br>
> + if (Variant.Refs)<br>
> + for (const auto &Ref : Variant.Refs->second)<br>
> + Refs.insert(Variant.Refs->first, Ref);<br>
> } while (Yin.nextDocument());<br>
> <br>
> IndexFileIn Result;<br>
> Result.Symbols.emplace(std::move(Symbols).build());<br>
> + Result.Refs.emplace(std::move(Refs).build());<br>
> return std::move(Result);<br>
> }<br>
> <br>
> @@ -217,6 +290,17 @@ std::string toYAML(const Symbol &S) {<br>
> }<br>
> return Buf;<br>
> }<br>
> +<br>
> +std::string toYAML(const std::pair<SymbolID, ArrayRef<Ref>> &Data) {<br>
> + RefBundle Refs = {Data.first, Data.second};<br>
> + std::string Buf;<br>
> + {<br>
> + llvm::raw_string_ostream OS(Buf);<br>
> + llvm::yaml::Output Yout(OS);<br>
> + Yout << Refs;<br>
> + }<br>
> + return Buf;<br>
> +}<br>
> <br>
> } // namespace clangd<br>
> } // namespace clang<br>
> <br>
> Modified: clang-tools-extra/trunk/clangd/indexer/IndexerMain.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/clang-tools-</a><br>
> extra/trunk/clangd/indexer/IndexerMain.cpp?rev=343778&r1=343777&r2=3437<br>
> 78&view=diff<br>
> =======================================================================<br>
> =======<br>
> --- clang-tools-extra/trunk/clangd/indexer/IndexerMain.cpp (original)<br>
> +++ clang-tools-extra/trunk/clangd/indexer/IndexerMain.cpp Thu Oct 4<br>
> 07:09:55 2018<br>
> @@ -67,18 +67,30 @@ public:<br>
> else<br>
> Symbols.insert(Sym);<br>
> }<br>
> + },<br>
> + [&](RefSlab S) {<br>
> + std::lock_guard<std::mutex> Lock(SymbolsMu);<br>
> + for (const auto &Sym : S) {<br>
> + // No need to merge as currently all Refs are from<br>
> main file.<br>
> + for (const auto &Ref : Sym.second)<br>
> + Refs.insert(Sym.first, Ref);<br>
> + }<br>
> })<br>
> .release();<br>
> }<br>
> <br>
> // Awkward: we write the result in the destructor, because the<br>
> executor<br>
> // takes ownership so it's the easiest way to get our data back out.<br>
> - ~IndexActionFactory() { Result.Symbols = std::move(Symbols).build();<br>
> }<br>
> + ~IndexActionFactory() {<br>
> + Result.Symbols = std::move(Symbols).build();<br>
> + Result.Refs = std::move(Refs).build();<br>
> + }<br>
> <br>
> private:<br>
> IndexFileIn &Result;<br>
> std::mutex SymbolsMu;<br>
> SymbolSlab::Builder Symbols;<br>
> + RefSlab::Builder Refs;<br>
> };<br>
> <br>
> } // namespace<br>
> <br>
> Modified: clang-tools-<br>
> extra/trunk/unittests/clangd/SerializationTests.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/clang-tools-</a><br>
> extra/trunk/unittests/clangd/SerializationTests.cpp?rev=343778&r1=34377<br>
> 7&r2=343778&view=diff<br>
> =======================================================================<br>
> =======<br>
> --- clang-tools-extra/trunk/unittests/clangd/SerializationTests.cpp<br>
> (original)<br>
> +++ clang-tools-extra/trunk/unittests/clangd/SerializationTests.cpp Thu<br>
> Oct 4 07:09:55 2018<br>
> @@ -13,6 +13,9 @@<br>
> #include "gmock/gmock.h"<br>
> #include "gtest/gtest.h"<br>
> <br>
> +using testing::_;<br>
> +using testing::AllOf;<br>
> +using testing::Pair;<br>
> using testing::UnorderedElementsAre;<br>
> using testing::UnorderedElementsAreArray;<br>
> namespace clang {<br>
> @@ -21,6 +24,7 @@ namespace {<br>
> <br>
> const char *YAML = R"(<br>
> ---<br>
> +!Symbol<br>
> ID: 057557CEBF6E6B2DD437FBF60CC58F352D1DF856<br>
> Name: 'Foo1'<br>
> Scope: 'clang::'<br>
> @@ -46,6 +50,7 @@ IncludeHeaders:<br>
> References: 3<br>
> ...<br>
> ---<br>
> +!Symbol<br>
> ID: 057557CEBF6E6B2DD437FBF60CC58F352D1DF858<br>
> Name: 'Foo2'<br>
> Scope: 'clang::'<br>
> @@ -64,6 +69,18 @@ Flags: 2<br>
> Signature: '-sig'<br>
> CompletionSnippetSuffix: '-snippet'<br>
> ...<br>
> +!Refs<br>
> +ID: 057557CEBF6E6B2DD437FBF60CC58F352D1DF856<br>
> +References:<br>
> + - Kind: 4<br>
> + Location:<br>
> + FileURI: file:///path/foo.cc<br>
> + Start:<br>
> + Line: 5<br>
> + Column: 3<br>
> + End:<br>
> + Line: 5<br>
> + Column: 8<br>
> )";<br>
> <br>
> MATCHER_P(ID, I, "") { return arg.ID ==<br>
> cantFail(SymbolID::fromStr(I)); }<br>
> @@ -107,6 +124,16 @@ TEST(SerializationTest, YAMLConversions)<br>
> EXPECT_EQ(Sym2.CanonicalDeclaration.FileURI, "file:///path/bar.h");<br>
> EXPECT_FALSE(Sym2.Flags & Symbol::IndexedForCodeCompletion);<br>
> EXPECT_TRUE(Sym2.Flags & Symbol::Deprecated);<br>
> +<br>
> + ASSERT_TRUE(bool(ParsedYAML->Refs));<br>
> + EXPECT_THAT(*ParsedYAML->Refs,<br>
> + UnorderedElementsAre(<br>
> + Pair(cantFail(SymbolID::fromStr(<br>
> +<br>
> "057557CEBF6E6B2DD437FBF60CC58F352D1DF856")),<br>
> + testing::SizeIs(1))));<br>
> + auto Ref1 = ParsedYAML->Refs->begin()->second.front();<br>
> + EXPECT_EQ(Ref1.Kind, RefKind::Reference);<br>
> + EXPECT_EQ(Ref1.Location.FileURI, "file:///path/foo.cc");<br>
> }<br>
> <br>
> std::vector<std::string> YAMLFromSymbols(const SymbolSlab &Slab) {<br>
> @@ -115,24 +142,37 @@ std::vector<std::string> YAMLFromSymbols<br>
> Result.push_back(toYAML(Sym));<br>
> return Result;<br>
> }<br>
> +std::vector<std::string> YAMLFromRefs(const RefSlab &Slab) {<br>
> + std::vector<std::string> Result;<br>
> + for (const auto &Sym : Slab)<br>
> + Result.push_back(toYAML(Sym));<br>
> + return Result;<br>
> +}<br>
> <br>
> TEST(SerializationTest, BinaryConversions) {<br>
> auto In = readIndexFile(YAML);<br>
> EXPECT_TRUE(bool(In)) << In.takeError();<br>
> <br>
> // Write to binary format, and parse again.<br>
> - IndexFileOut Out;<br>
> - Out.Symbols = In->Symbols.getPointer();<br>
> + IndexFileOut Out(*In);<br>
> Out.Format = IndexFileFormat::RIFF;<br>
> std::string Serialized = llvm::to_string(Out);<br>
> + {<br>
> + std::error_code EC;<br>
> + llvm::raw_fd_ostream F("/tmp/foo", EC);<br>
> + F << Serialized;<br>
> + }<br>
> <br>
> auto In2 = readIndexFile(Serialized);<br>
> ASSERT_TRUE(bool(In2)) << In.takeError();<br>
> - ASSERT_TRUE(In->Symbols);<br>
> + ASSERT_TRUE(In2->Symbols);<br>
> + ASSERT_TRUE(In2->Refs);<br>
> <br>
> // Assert the YAML serializations match, for nice comparisons and<br>
> diffs.<br>
> EXPECT_THAT(YAMLFromSymbols(*In2->Symbols),<br>
> UnorderedElementsAreArray(YAMLFromSymbols(*In-<br>
> >Symbols)));<br>
> + EXPECT_THAT(YAMLFromRefs(*In2->Refs),<br>
> + UnorderedElementsAreArray(YAMLFromRefs(*In->Refs)));<br>
> }<br>
> <br>
> } // namespace<br>
> <br>
> <br>
> _______________________________________________<br>
> cfe-commits mailing list<br>
> <a href="mailto:cfe-commits@lists.llvm.org" target="_blank">cfe-commits@lists.llvm.org</a><br>
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits</a><br>
</blockquote></div>