<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On 11 August 2017 at 17:09, Doug Schaefer via cfe-dev <span dir="ltr"><<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div class="m_7013316755050223983WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Sorry gang, Outlook isn’t treating this thread very well. I’ll have to top post my response.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">The CDT indexer and database were written in 2005 so the data is long gone and we’re talking about different database technologies anyway. All I remember was it was slower
by multiple orders of magnitude on writes. And that stopped my investigation right there. At the time I was trying to get a full index of the Firefox source down from an hour to a few minutes and incremental indexing times down to less than a second (thanks
to the header caching). The DB was taking me in the other direction.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I’m just not sure architecting for simple lookups is the right approach. Just consider probably the most common use case for clangd, content assist. Given an offset into the
file, determine the context/scope of the content and a prefix. You then need to find all matches for that prefix. That’s by no means a simple lookup given all the declarations, type hierarchies, scope hierarchies in play in that context. Finding references
is another good one that also requires scope analysis to make sure you’re actually returning references to that exact definition and not just something that happens to have the same name. The CDT index has links back and forth between these things.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u></span></p></div></div></blockquote><div><br></div><div>Clang shouldn't query by name when looking up references, instead USRs should be used. The USRs deal with scopes correctly (Hence the U in USR). </div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div class="m_7013316755050223983WordSection1"><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Again, I’m just relaying my experience haven built one of these things before. And it was a long time ago. All avenues are worth exploring.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Doug.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Manuel Klimek [mailto:<a href="mailto:klimek@google.com" target="_blank">klimek@google.com</a>]
<br>
<b>Sent:</b> Friday, August 11, 2017 10:59 AM</span></p><div><div class="h5"><br>
<b>To:</b> Doug Schaefer <<a href="mailto:dschaefer@blackberry.com" target="_blank">dschaefer@blackberry.com</a>>; David Chisnall <<a href="mailto:David.Chisnall@cl.cam.ac.uk" target="_blank">David.Chisnall@cl.cam.ac.uk</a>>; Marc-André Laperle <<a href="mailto:marc-andre.laperle@ericsson.com" target="_blank">marc-andre.laperle@ericsson.<wbr>com</a>><br>
<b>Cc:</b> via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>><br>
<b>Subject:</b> Re: [cfe-dev] Adding indexing support to Clangd<u></u><u></u></div></div><p></p>
</div>
</div><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">On Fri, Aug 11, 2017 at 4:50 PM Doug Schaefer <<a href="mailto:dschaefer@blackberry.com" target="_blank">dschaefer@blackberry.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Manuel Klimek [mailto:<a href="mailto:klimek@google.com" target="_blank">klimek@google.com</a>]
<br>
<b>Sent:</b> Friday, August 11, 2017 4:54 AM<br>
<b>To:</b> Doug Schaefer <<a href="mailto:dschaefer@blackberry.com" target="_blank">dschaefer@blackberry.com</a>>; David Chisnall <<a href="mailto:David.Chisnall@cl.cam.ac.uk" target="_blank">David.Chisnall@cl.cam.ac.uk</a>>; Marc-André Laperle <<a href="mailto:marc-andre.laperle@ericsson.com" target="_blank">marc-andre.laperle@ericsson.<wbr>com</a>><br>
<b>Cc:</b> via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>></span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
<b>Subject:</b> Re: [cfe-dev] Adding indexing support to Clangd</span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div>
<div>
<p class="MsoNormal">On Thu, Aug 10, 2017 at 5:57 PM Doug Schaefer via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal">> -----Original Message-----<br>
> From: Dr D. Chisnall [mailto:<a href="mailto:dc552@hermes.cam.ac.uk" target="_blank">dc552@hermes.cam.ac.uk</a><wbr>] On Behalf Of David<br>
> Chisnall<br>
> Sent: Thursday, August 10, 2017 6:10 AM<br>
> To: Marc-André Laperle <<a href="mailto:marc-andre.laperle@ericsson.com" target="_blank">marc-andre.laperle@ericsson.<wbr>com</a>><br>
> Cc: via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>>;
<a href="mailto:zeratul976@hotmail.com" target="_blank">zeratul976@hotmail.com</a>; Doug<br>
> Schaefer <<a href="mailto:dschaefer@blackberry.com" target="_blank">dschaefer@blackberry.com</a>><br>
> Subject: Re: [cfe-dev] Adding indexing support to Clangd<br>
><br>
> On 8 Aug 2017, at 18:52, Marc-André Laperle via cfe-dev <cfe-<br>
> <a href="mailto:dev@lists.llvm.org" target="_blank">dev@lists.llvm.org</a>> wrote:<br>
> ><br>
> > --ClangdIndexStorage--<br>
> > malloc-like interface that allocates/frees data blocks of variable sizes on<br>
> disk. The blocks can contain ints, shorts, strings, pointers (i.e. file offsets),<br>
> etc. The data is cached in 4K pieces so that local and repeated accesses are all<br>
> done quickly, in memory.<br>
> > Clangd mallocs and writes its index model objects using this.<br>
> ><br>
> > --BTree--<br>
> > A pretty classic BTree implementation. Used to speed up lookup (symbol<br>
> names, file names). It allocates its nodes using ClangdIndexStorage therefore<br>
> it is stored on disk. Keys are actually records in ClangdIndexStorage so you<br>
> can really think of the BTree as a collection of sorted pointers (sorted<br>
> according to a provided comparator).<br>
><br>
> This sounds very like bdb. Is there a reason that we’re reimplementing a<br>
> large chunk of bdb, rather than just using it (or using something like sqlite for<br>
> the index storage)?<br>
<br>
It looks like Marc-Andre is following our work on the Eclipse CDT indexer. We found databases to be just too slow.<u></u><u></u></p>
</blockquote>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">That is somewhat surprising. Which db's did you benchmark against?<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">This was many years ago and we were trying Apache Derby (since Eclipse is written in Java). And at the time the biggest
issue was write performance totally destroying any gains we made with header caching. But that may have just been Derby.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Though I think the bigger issue is that the language model is a graph which doesn’t fit naturally in tables. You’ll
end up spending more time fighting that than building out your own data structures. But that’s just my experience.</span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Yea, the language model not fitting into tables well does match my intuition, but for simple queries, I'd have expected a db to be fast enough with room to spare. Would be curious about more details / numbers.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal">Also, forcing the C++ language model to fit into a bunch of tables or key/value pairs just didn't make that much sense in the end. The data is very complex.<br>
<br>
Instead it's better to think of this as a big graph data structure that just happens to be backed by memory mapped file store. Clients generally deal directly with pointers into that data structure and follow pointers in the graph. The BTrees help speed up
that navigation at various nodes in the graph. It's super fast.<br>
<br>
Doug.<br>
______________________________<wbr>_________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/cfe-dev</a><u></u><u></u></p>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div></div></div>
</div>
</div>
<br>______________________________<wbr>_________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/cfe-dev</a><br>
<br></blockquote></div><br></div></div>