[cfe-dev] RFC: Adding index-while-building support to Clang

[Argyrios Kyrtzidis via cfe-dev]

Hi Ilya,

> On Aug 29, 2017, at 1:55 AM, Ilya Biryukov via cfe-dev <cfe-dev at lists.llvm.org> wrote:
> 
> Hi,
> 
> Thanks for the design.
> 
> 1. You briefly mention an in-memory mapping on top the index to support efficient lookup of index records.
> Is it part of the available implementation? Do you think it would make sense to reuse it in other tools?
> 
> I'm asking because that seems like something most editors and IDEs are also interested in. Proper reuse of this implementation could prove useful.

In our implementation we use LMDB (https://symas.com/lightning-memory-mapped-database). It is a key-value data-store that we use for cross-referencing queries, similarly to the example that Nathan provides in the document.

Is this something that we could accept into the clang project (e.g. in clang-tools-extra) ? Note it is essentially a single header and implementation file.

> 
> 2.  In clangd, we're not controlling the build step, instead building ASTs in-memory. We would rather store the indexing information in-memory or consume it on the go while building ASTs.
> Do you have suggestions on which parts of the API we should look at?
> We could implement our own IndexASTConsumer, but are there more opportunities for reusing other parts of your implementation? Code for collecting indexing dependencies, definitions of high-level record structures (i.e. symbol definitions, etc.)?

There are a few ways to go about this:

- Have ASTs in-memory, but indexing works on the file system. It’s not ideal but it is simple and works fairly well in practice, particularly since in our platform, files open in Xcode can be saved in disk even without having the user explicitly saving them.
- Update clang’s raw index data store using the in-memory buffers and ASTs. The simplicity is that symbol info comes from one place only, but there’s complexity in that you have raw data on disk that reflect in-memory-only sources.
- The layer on-top of clang's raw index data store is enhanced to treat the raw data on-disk as one source of symbol info, and in-memory ASTs as another. For example, if using LMDB, you could have it distinguish that info about a symbol comes from the raw data on-disk vs an in-memory AST.

> 
> 
> On Tue, Aug 29, 2017 at 9:34 AM, Manuel Klimek <klimek at google.com <mailto:klimek at google.com>> wrote:
> Hi, thanks for the details design, I'm really looking forward for having this :)
> 
> On a high level, I think this all makes lots of sense. 
> I have one question about how things get deleted: will record files for headers be deleted at some point, or will they be kept around, and you have to look at all unit files to see whether a given record file is still valid?
> 
> Cheers,
> /Manuel
> 
> 
> On Tue, Aug 29, 2017 at 2:18 AM Nathan Hawes <nhawes at apple.com <mailto:nhawes at apple.com>> wrote:
> Hey everyone,
> 
> Xcode 9 shipped with index-while-building functionality based on enhancements to Clang that we’d like to upstream. Key among them is a new option, -index-store-path, that in addition to Clang's usual outputs, causes it to write out indexing data at the supplied path with minimal overhead.
> 
> While the current implementation is available at https://github.com/apple/swift-clang <https://github.com/apple/swift-clang>, we’d like to start by getting feedback on the high-level design, which you can read about here: https://docs.google.com/document/d/1cH2sTpgSnJZCkZtJl1aY-rzy4uGPcrI-6RrUpdATO2Q/edit?usp=sharing <https://docs.google.com/document/d/1cH2sTpgSnJZCkZtJl1aY-rzy4uGPcrI-6RrUpdATO2Q/edit?usp=sharing>
> 
> Please let us know of any concerns, comments or questions you have regarding this feature.
> 
> Thanks!
> Nathan
> 
> 
> 
> -- 
> Regards,
> Ilya Biryukov
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20170830/41ffbaa5/attachment.html>