[cfe-dev] Adding indexing support to Clangd

Thu May 25 17:07:01 PDT 2017

Hello everyone,

On 23.05.2017 07:02, Marc-André Laperle via cfe-dev wrote:
>
> Those are good points. I think we'll have to see how fast the indexing 
> without caching and then go from there.
>
We did such measurements when evaluated clang as a technology to be used 
in NetBeans C/C++, I don't remember the exact absolute numbers now, but 
the conclusion was:
to be on par with the existing NetBeans speed we have to use different 
caching, otherwise it was like 10 times slower.

> I think the precompiled preamble sounds useful at least for opened 
> editors for quick parsing as there can be many reparsings of the same 
> file without anything changes in the inclusions (while typing, etc).
>
+1. Preambles need to be used to provides reasonable responsiveness. 
Otherwise i.e. included boost header can consume unexpected time.
In fact sometimes it's worth to have preambles granularity per-functions 
(i.e. for files opened in editor), because when developers modify code 
then most of the time they modify bodies.

> I think reusing the information from the previous version is a good 
> compromise but we also have to make sure building the index for the 
> first time is not too long or at least make sure that there is 
> sufficient functionality available for users to start working and 
> communicate that some functionality is not available.
>
+1. Btw, may be It is worth to set some expectations what is available 
during and after initial index phase.
I.e. during initial phase you'd probably like to have navigation for 
file opened in editor and can work in functions bodies.

As to initial indexing:
Using PTH (not PCH) gave significant speedup.
Skipping bodies gave significant speedup, but you miss the references 
and later have to reindex bodies on demand.
Using chainged PCH gave the next visible speedup.

Of course we had to made some hacks for PCHs to be more often "reusable" 
(comparing to strict compiler rule) and keep multiple versions. In 
average 2: one for C and one for C++ parse context.

Also there is a difference between system headers and projects headers, 
so systems' can be cached more aggressively.

Vladimir
NetBeans C/C++ Project Lead

>
> Cheers,
>
> Marc-André
>
> ------------------------------------------------------------------------
> *From:* Ilya Biryukov <ibiryukov at google.com>
> *Sent:* Friday, May 19, 2017 8:27:48 AM
> *To:* Marc-André Laperle
> *Cc:* Doug Schaefer; via cfe-dev; zaks.anna at gmail.com; Zoltan 
> Porkoláb; Marton Csordas
> *Subject:* Re: [cfe-dev] Adding indexing support to Clangd
> Hi everyone,
>
> The problem with PCHs(either chained or not) is that they only work 
> for the source file,
> i.e. you can only use it when you start the new file from scratch, 
> right? For header caching we
> really want to reuse whatever information we have cached even if it's 
> included in a different
> context(i.e. the order of includes is different in the other 
> translation unit), which is not possible with PCHs.
>
> My point is that it's not at all straightforward how(or if?) the PCHs 
> can improve performance of processing
> the same header twice.
>
> And as long as building an index is as fast as a recompile and we can 
> reuse information from the previous
> version(slightly outdated) the index while the new version is 
> building, we can probably get a good enough
> UX without any compromises on correctness(and introducing additional 
> complexity, since I don't think there's
> a way to do header caching without significant changes to clang itself).
>
> CLion indeed has a custom parser and serialization format, it's not 
> clang-based.
>
>
> On Thu, May 18, 2017 at 10:33 PM, Marc-André Laperle via cfe-dev 
> <cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>> wrote:
>
>     Yeah, it sounds like a good approach to tackle header caching a
>     bit later. "Precompiled preamble" looks promising so we can keep
>     this in mind as we go.
>
>
>     Cheers,
>
>     Marc-André
>
>     ------------------------------------------------------------------------
>     *From:* Doug Schaefer <dschaefer at blackberry.com
>     <mailto:dschaefer at blackberry.com>>
>     *Sent:* Thursday, May 18, 2017 10:44:18 AM
>     *To:* Marc-André Laperle; via cfe-dev
>     *Cc:* zeratul976 at hotmail.com <mailto:zeratul976 at hotmail.com>;
>     zaks.anna at gmail.com <mailto:zaks.anna at gmail.com>; Dániel Krupp;
>     Zoltan Porkoláb; Marton Csordas
>     *Subject:* Re: Adding indexing support to Clangd
>     On 2017-05-17, 6:38 PM, "Marc-André Laperle"
>     <marc-andre.laperle at ericsson.com
>     <mailto:marc-andre.laperle at ericsson.com>> wrote:
>     >  - For header caching, I wonder if it is possible to reuse the
>     >precompiled header support in Clang. There would be some logic
>     that would
>     >decide whether or not a precompiled header could be used
>     depending on the
>     >preprocessing context (same macro definitions, etc).
>
>     I¹m not certain header cacheing is needed right away. We did it in
>     the CDT
>     because our parsers were fairly slow and indexing a project took a
>     very
>     long time. I have hope that clang would be faster. At the very
>     least, you
>     would want cacheing to be optional so you need to be able to work
>     without
>     it. But make sure you have the architecture to graft it in later.
>
>     In CDT we cheated a lot to gain performance and the cost of
>     accuracy. The
>     results are still very good so its an interesting balancing act.
>
>     Doug Schaefer
>     Eclipse CDT Project Lead
>
>
>     _______________________________________________
>     cfe-dev mailing list
>     cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>
>     http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>     <http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev>
>
>
>
>
> -- 
> Regards,
> Ilya Biryukov
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20170526/feb14b35/attachment.html>