<div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Fri, Aug 11, 2017 at 4:50 PM Doug Schaefer <<a href="mailto:dschaefer@blackberry.com">dschaefer@blackberry.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div class="m_3838757905106044830WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Manuel Klimek [mailto:<a href="mailto:klimek@google.com" target="_blank">klimek@google.com</a>]
<br>
<b>Sent:</b> Friday, August 11, 2017 4:54 AM<br>
<b>To:</b> Doug Schaefer <<a href="mailto:dschaefer@blackberry.com" target="_blank">dschaefer@blackberry.com</a>>; David Chisnall <<a href="mailto:David.Chisnall@cl.cam.ac.uk" target="_blank">David.Chisnall@cl.cam.ac.uk</a>>; Marc-André Laperle <<a href="mailto:marc-andre.laperle@ericsson.com" target="_blank">marc-andre.laperle@ericsson.com</a>><br>
<b>Cc:</b> via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>></span></p></div></div></div></div></div><div lang="EN-US" link="blue" vlink="purple"><div class="m_3838757905106044830WordSection1"><div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt"><div><div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
<b>Subject:</b> Re: [cfe-dev] Adding indexing support to Clangd<u></u><u></u></span></p></div></div></div></div></div><div lang="EN-US" link="blue" vlink="purple"><div class="m_3838757905106044830WordSection1"><div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt"><div><div>
<div>
<p class="MsoNormal">On Thu, Aug 10, 2017 at 5:57 PM Doug Schaefer via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">> -----Original Message-----<br>
> From: Dr D. Chisnall [mailto:<a href="mailto:dc552@hermes.cam.ac.uk" target="_blank">dc552@hermes.cam.ac.uk</a>] On Behalf Of David<br>
> Chisnall<br>
> Sent: Thursday, August 10, 2017 6:10 AM<br>
> To: Marc-André Laperle <<a href="mailto:marc-andre.laperle@ericsson.com" target="_blank">marc-andre.laperle@ericsson.com</a>><br>
> Cc: via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>>;
<a href="mailto:zeratul976@hotmail.com" target="_blank">zeratul976@hotmail.com</a>; Doug<br>
> Schaefer <<a href="mailto:dschaefer@blackberry.com" target="_blank">dschaefer@blackberry.com</a>><br>
> Subject: Re: [cfe-dev] Adding indexing support to Clangd<br>
><br>
> On 8 Aug 2017, at 18:52, Marc-André Laperle via cfe-dev <cfe-<br>
> <a href="mailto:dev@lists.llvm.org" target="_blank">dev@lists.llvm.org</a>> wrote:<br>
> ><br>
> > --ClangdIndexStorage--<br>
> > malloc-like interface that allocates/frees data blocks of variable sizes on<br>
> disk. The blocks can contain ints, shorts, strings, pointers (i.e. file offsets),<br>
> etc. The data is cached in 4K pieces so that local and repeated accesses are all<br>
> done quickly, in memory.<br>
> > Clangd mallocs and writes its index model objects using this.<br>
> ><br>
> > --BTree--<br>
> > A pretty classic BTree implementation. Used to speed up lookup (symbol<br>
> names, file names). It allocates its nodes using ClangdIndexStorage therefore<br>
> it is stored on disk. Keys are actually records in ClangdIndexStorage so you<br>
> can really think of the BTree as a collection of sorted pointers (sorted<br>
> according to a provided comparator).<br>
><br>
> This sounds very like bdb. Is there a reason that we’re reimplementing a<br>
> large chunk of bdb, rather than just using it (or using something like sqlite for<br>
> the index storage)?<br>
<br>
It looks like Marc-Andre is following our work on the Eclipse CDT indexer. We found databases to be just too slow.<u></u><u></u></p>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">That is somewhat surprising. Which db's did you benchmark against?<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
</div></div></div></div></div><div lang="EN-US" link="blue" vlink="purple"><div class="m_3838757905106044830WordSection1"><div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt"><div><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">This was many years ago and we were trying Apache Derby (since Eclipse is written in Java). And at the time the biggest issue was write performance totally destroying any gains
we made with header caching. But that may have just been Derby.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Though I think the bigger issue is that the language model is a graph which doesn’t fit naturally in tables. You’ll end up spending more time fighting that than building out
your own data structures. But that’s just my experience.</span></p></div></div></div></div></div></blockquote><div><br></div><div>Yea, the language model not fitting into tables well does match my intuition, but for simple queries, I'd have expected a db to be fast enough with room to spare. Would be curious about more details / numbers.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div class="m_3838757905106044830WordSection1"><div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt"><div><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u><u></u></span></p></div></div></div></div></div><div lang="EN-US" link="blue" vlink="purple"><div class="m_3838757905106044830WordSection1"><div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt"><div><div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Also, forcing the C++ language model to fit into a bunch of tables or key/value pairs just didn't make that much sense in the end. The data is very complex.<br>
<br>
Instead it's better to think of this as a big graph data structure that just happens to be backed by memory mapped file store. Clients generally deal directly with pointers into that data structure and follow pointers in the graph. The BTrees help speed up
that navigation at various nodes in the graph. It's super fast.<br>
<br>
Doug.<br>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><u></u><u></u></p>
</blockquote>
</div></div></div></div></div></blockquote></div></div>