[cfe-dev] Libclang: Bad performance on reparse

[John McCall via cfe-dev]

> On Aug 16, 2017, at 4:18 AM, Janosch Machowinski via cfe-dev <cfe-dev at lists.llvm.org> wrote:
> Hi,
> while testing kdevelop 5, I ran into some problems with the code completion.
> I debugged it and tracked it down to the reparse step in libclang taking
> 5 seconds in an i7-5930K.
> 
> In my concrete example, I got a cpp file that consists of 173 line of code,
> with minor template usage.
> This program includes a big amount of headers, that use heavy templating.

Well, if the template usage in the headers is mostly dependent, PCH alone isn't going
to help much.  We may be able to see that A<T> is using B<T> and C<T>, but we can't
actually instantiate any of that until we get a request for a specialization at a concrete
type, which we may not see until we get to the main source file.

There are a number of things that can be done on top of a PCH system to help move
some of that computation time into PCH generation.  For example, the tool could remember
the set of types that the templates are instantiated at outside of the PCH and then
re-generate the PCH with eager instantiations.  Also, of course, the tool may not actually
require those instantiations to be done at all; it may benefit from a great deal more laziness.
But all of this requires more logic from the tool, not just generating a PCH.

That may not be what's happening here, of course.

John.

> 
> The initial parse takes 13 seconds, which is fine. What makes me wonder is
> the reparse step taking so long, as everything in the headers should be
> cached by the PCH.
> 
> I prepared a git repo with an example, if you want to look at the code I used,
> or try it yourself.
> 
> https://github.com/jmachowinski/clangTest
> 
> It should be noted, that I do not set CXTranslationUnit_Incomplete,
> as kdevelop needs the flag to be NOT set. Setting the flag speeds up reparse
> to 0.5 seconds.
> 
> Greetings
>    Janosch
> 
> 
> Debug output :
> 
> Parse start
> Precompiling preamble:   8.2960 (100.0%)   0.2760 (100.0%)   8.5720 (100.0%)   8.5706 (100.0%)
> Parsing ../test/tdot_asguard.cpp:   4.8320 (100.0%)   0.0200 (100.0%)   4.8520 (100.0%)   4.8499 (100.0%)
>  ASTContext: expressions, declarations, and types: 254289451
>  ASTContext: identifiers: 507904
>  ASTContext: selectors: 4096
>  ASTContext: side tables: 9053176
>  Code completion: cached global results: 0
>  SourceManager: content cache allocator: 86016
>  SourceManager: malloc'ed memory buffers: 2070147
>  SourceManager: mmap'ed memory buffers: 0
>  SourceManager: data structures and tables: 124472904
>  ExternalASTSource: malloc'ed memory buffers: 0
>  ExternalASTSource: mmap'ed memory buffers: 188282552
>  Preprocessor: malloc'ed memory: 306782
>  Preprocessor: PreprocessingRecord: 838904
>  Preprocessor: header search tables: 43
> Parse took 13.4559
> Cache global code completions for ../test/tdot_asguard.cpp:   0.1280 (100.0%)   0.1280 (100.0%)   0.1267 (100.0%)
> Reparsing ../test/tdot_asguard.cpp:   5.0600 (100.0%)   0.0440 (100.0%)   5.1040 (100.0%)   5.1043 (100.0%)
>  ASTContext: expressions, declarations, and types: 259532331
>  ASTContext: identifiers: 2326528
>  ASTContext: selectors: 4096
>  ASTContext: side tables: 9151480
>  Code completion: cached global results: 4063232
>  SourceManager: content cache allocator: 111147
>  SourceManager: malloc'ed memory buffers: 2082327
>  SourceManager: mmap'ed memory buffers: 0
>  SourceManager: data structures and tables: 124472904
>  ExternalASTSource: malloc'ed memory buffers: 0
>  ExternalASTSource: mmap'ed memory buffers: 188282552
>  Preprocessor: malloc'ed memory: 8324702
>  Preprocessor: PreprocessingRecord: 1775864
>  Preprocessor: header search tables: 43
> ReParse took 5.10452
> 
> 
> 
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev