[cfe-dev] Unbundling key parts of libIndex/IndexingContext

[Sam McCall via cfe-dev]

TL;DR: I'd really like to extract the part that goes from AST node ->
Decl + Role, but I don't know how to do it incrementally. Current plan
would be to copy it to something parallel, and maybe try to port libIndex
later. That sounds likely to end in permanent duplication though. Help!

--

Hi tooling folks,

There are several pieces in libIndex that are important for indexing, but
also useful for other things. clangd (and I would guess libclang) has been
(ab)using these, e.g. implementing "go to definition" by indexing the file
and looking for decl references that overlap the cursor.
But performance, flexibility, and understanding tend to suffer, as the APIs
don't always suit the problem space.

I think IndexingAction/IndexingContext do the following things:
A) expose macros while parsing/recorded in PP through a common interface
B) traverse the AST (according to some policy) to find AST nodes
C) determine which tokens in the source refer to those nodes
D) determine which decls AST nodes refer to, and with which roles
E) find decls related to an AST node (e.g. overrides)
There's also F) getSymbolInfo(), which is already split out.

These seem mostly[1] disjoint to me, and can each be replaced/dropped in
some scenarios. examples:
- finding AST nodes (B) is sometimes better done with
ASTSelection/SelectionTree/RecursiveASTVisitor.
- hover-to-evaluate-constant needs C but not D
- analyses like extract-variable care about D but not C or E

AST Node -> Decl is the largest and most useful chunk. I've built a
standalone prototype which does pretty well on C++, and porting more of the
logic from libIndex could bring it up to par.
The API is something like
  vector<pair<const Decl*, SymbolRoleSet>> findDecl(DynTypedNode,
     ReferencePolicy, // lexical references only, or semantic ones like
`decltype(...)` too?
     AliasPolicy, // for typedefs etc, return the alias, the underlying
decl, or both?
     TemplatePolicy, // return specialization, the relevant pattern, the
primary template, or some combination?
  )
The Policy options allow us to more directly express features and fix some
bugs.

My main concern is that I'm not sure how to eventually (ideally
incrementally!) use this in libIndex.
The natural implementation is a set of mutually recursive functions to
handle Decl/Type/Stmt etc (much like libIndex's structure), so before
calling any of them, all must be complete and correct.
And SymbolRelations seem to sometimes need information from the AST
traversal, and sometimes information from the node->decl walk, so it's hard
to see how they can be separated.

So though it seems insane to have two copies of this logic, I find myself
leaning toward a separate implementation. Is there a better way out than
this? How much value is there in putting it in Tooling vs locally in clangd?

Cheers, Sam

[1] B and E seem fairly tangled, actually.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20190809/18516936/attachment.html>