[PATCH] D89749: SourceManager: Don't allocate an SLocEntry until it's loaded

[Duncan P. N. Exon Smith via Phabricator via cfe-commits]

dexonsmith added a comment.

In D89749#2356846 <https://reviews.llvm.org/D89749#2356846>, @v.g.vassilev wrote:

> Is the performance overhead for loaded module a % of the overall size of the source files?

`SourceManager::PrintStats` almost has the information you need. One of the lines is:

  llvm::errs() << LoadedSLocEntryTable.size()
                 << " loaded SLocEntries allocated, "

This is how many `SLocEntries` have been allocated for imported ASTs (modules / PCH). But unless the corresponding bit in `SLocEntryLoaded` is set, it hasn't actually been "loaded", a spot has just been reserved. The first call to `getSLocEntry` for that location will trigger a load from the serialized AST.

Here's how you can estimate the memory usage change:

  uint64_t OldMem = 0, NewMem = 0;
  
  // Add each allocated SLocEntry. NextPowerOf2 is an estimate of the SmallVector
  // load factor.
  OldMem += llvm::capacity_in_bytes(LoadedSLocEntryTable);
  NewMem += NextPowerOf2(SLocEntryTableLoaded.count()) * sizeof(SLocEntry);
  
  // Add the side table overhead. We know precisely what the load factor of
  // is in this case, since we have one that's the same size.
  OldMem += llvm::capacity_in_bytes(SLocEntryTableLoaded);
  NewMem += LoadedSLocEntryTable.capacity() * sizeof(unsigned);

If you do that calculation at the end of the compilation, you can see the memory savings.

> Sorry for the naive question but what is a unloaded module?

It's an unloaded `SLocEntry`. After the module has been imported, the `SLocEntry`s are allocated, but lazily read from disk on first access. If it hasn't been read from disk yet, I'm calling it "unloaded".



================
Comment at: clang/include/clang/Basic/SourceManager.h:703
-  /// Same indexing as LoadedSLocEntryTable.
-  llvm::BitVector SLocEntryLoaded;
 
----------------
v.g.vassilev wrote:
> We can probably remove the bitvector header include.
Nice catch, I'll fix that.


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D89749/new/

https://reviews.llvm.org/D89749