[clang] Introduce paged vector (PR #66430)

Vassil Vassilev via cfe-commits cfe-commits at lists.llvm.org
Fri Sep 15 03:10:07 PDT 2023


vgvassilev wrote:

> @ChuanqiXu9 I have added a simple unit test, which has 100% code coverage as far as I can tell and covers the usecase the container will be used for.
> 
> @nikic This is in the context of improving the memory used when loading modules, in particular my usecase is the runtime overhead of the ROOT/cling C++ interpreter / I/O subsystem.
> 
> Without this patch, the initialisation of the system, which on startup loads a bunch of unneeded modules, grows to up 80 MB as you can see from:
> 
> ![image](https://user-images.githubusercontent.com/10544/268231067-9c2c7dc3-8e64-403c-a923-97a40cd8f1ff.png)
> 
> With this patch applied, the total overhead goes down to 52MB:
> 
> ![image](https://user-images.githubusercontent.com/10544/268231334-56111b37-fa0e-415d-8d00-0bcd818db3aa.png)
> 
> As you can see the `ROOT::Internal::GetROOT2()` initialisation method goes from 67 to 30 MB (with some of the allocations which are now moved later on, due to the lazy allocation of the PagedVector).

@ktf, could you update the description of the PR including details about the reasoning of this change and the current benchmarks.

@nikic, this work is in the context of https://github.com/root-project/root/issues/13000. 

TL;DR: The downstream client has a long running session and preloads a relatively large list of modules. The clang::ASTReader does almost all of of the work lazily when a declaration is needed. However, in a few places we pre-allocate large vectors (for types or source locations for the entire module file) just to make  sure the sequencing is right. While this patch will not make this lazy, as we should, it goes a long way towards making this a relatively cheap for module content which is sparsely used. One good example where this easily becomes a problem is building a pcm file for boost.

https://github.com/llvm/llvm-project/pull/66430


More information about the cfe-commits mailing list