[PATCH] D99110: [Coverage] Load records immediately

[Choongwoo Han via Phabricator via llvm-commits]

tunz added a comment.

Here is the memory profile. I couldn't take a snapshot at the peak RSS, so I took a snapshot at the first 6GB.
The followings are the top 5 heap objects and their callstack.

- [1] void (size: 2.5GB) (count: 8,889,503)
- [2] char[] (size: 1GB) (count: 2)
- [3] char[] (size: 777MB) (count: 11)
- [4] llvm-cov.exe!llvm::coverage::BinaryCoverageReader::ProfileMappingRecord[]	(size: 300MB) (count: 10)
- [5] llvm-cov.exe!std::pair<unsigned __int64,llvm::StringRef>[] (size: 278MB) (count: 11)

[1]

  llvm-cov.exe!operator new() - Line 35
  llvm-cov.exe!llvm::allocate_buffer() - Line 21
  llvm-cov.exe!llvm::MallocAllocator::Allocate() - Line 86
  llvm-cov.exe!llvm::StringMapEntry<enum llvm::NoneType>::Create<llvm::MallocAllocator>() - Line 100
  llvm-cov.exe!llvm::StringMap<enum llvm::NoneType,llvm::MallocAllocator>::try_emplace<>() - Line 324
  llvm-cov.exe!llvm::StringSet<llvm::MallocAllocator>::insert() - Line 34
  llvm-cov.exe!llvm::InstrProfSymtab::addFuncName() - Line 483
  llvm-cov.exe!llvm::readPGOFuncNameStrings() - Line 494
  llvm-cov.exe!llvm::InstrProfSymtab::create() - Line 538
  llvm-cov.exe!`anonymous namespace'::CovMapFuncRecordReader::get<unsigned __int64,1>() - Line 751
  llvm-cov.exe!readCoverageMappingData<unsigned __int64,1>() - Line 788
  llvm-cov.exe!llvm::coverage::BinaryCoverageReader::createCoverageReaderFromBuffer() - Line 837
  llvm-cov.exe!loadBinaryFormat() - Line 986

or

  llvm-cov.exe!llvm::StringMapImpl::RehashTable() - Line 219
  llvm-cov.exe!llvm::readPGOFuncNameStrings() - Line 494

[2]

  llvm-cov.exe!operator new() - Line 30
  llvm-cov.exe!llvm::WritableMemoryBuffer::getNewUninitMemBuffer() - Line 298
  llvm-cov.exe!getOpenFileImpl<llvm::MemoryBuffer>() - Line 470
  llvm-cov.exe!getFileAux<llvm::MemoryBuffer>() - Line 268
  llvm-cov.exe!llvm::MemoryBuffer::getFile() - Line 247
  llvm-cov.exe!llvm::MemoryBuffer::getFileOrSTDIN() - Line 151
  llvm-cov.exe!llvm::coverage::CoverageMapping::load() - Line 324

[3]

  llvm-cov.exe!operator new() - Line 35
  llvm-cov.exe!std::_Default_allocate_traits::_Allocate() - Line 78
  llvm-cov.exe!std::_Allocate_manually_vector_aligned<std::_Default_allocate_traits>() - Line 120
  llvm-cov.exe!std::_Allocate<16,std::_Default_allocate_traits,0>() - Line 201
  llvm-cov.exe!std::allocator<char>::allocate() - Line 815
  llvm-cov.exe!std::basic_string<char,std::char_traits<char>,std::allocator<char> >::_Reallocate_grow_by<<lambda_65e615be2a453ca0576c979606f46740>,char const *,unsigned __int64>() - Line 4341
  llvm-cov.exe!std::basic_string<char,std::char_traits<char>,std::allocator<char> >::append() - Line 2955
  llvm-cov.exe!llvm::operator+=() - Line 924
  llvm-cov.exe!loadBinaryFormat() - Line 981

[4]

  llvm-cov.exe!operator new() - Line 35
  llvm-cov.exe!std::_Allocate_manually_vector_aligned<std::_Default_allocate_traits>() - Line 120
  llvm-cov.exe!`anonymous namespace'::VersionedCovMapFuncRecordReader<5,unsigned __int64,1>::insertFunctionRecordIfNeeded() - Line 565
  llvm-cov.exe!`anonymous namespace'::VersionedCovMapFuncRecordReader<5,unsigned __int64,1>::readFunctionRecords() - Line 722
  llvm-cov.exe!readCoverageMappingData<unsigned __int64,1>() - Line 810
  llvm-cov.exe!llvm::coverage::BinaryCoverageReader::createCoverageReaderFromBuffer() - Line 838
  llvm-cov.exe!loadBinaryFormat() - Line 986

[5]

  llvm-cov.exe!operator new() - Line 35
  llvm-cov.exe!std::_Allocate_manually_vector_aligned<std::_Default_allocate_traits>() - Line 120
  llvm-cov.exe!std::vector<std::pair<unsigned __int64,llvm::StringRef>,std::allocator<std::pair<unsigned __int64,llvm::StringRef> > >::_Emplace_reallocate<std::pair<unsigned __int64,llvm::StringRef> >() - Line 749
  llvm-cov.exe!llvm::readPGOFuncNameStrings() - Line 494
  llvm-cov.exe!`anonymous namespace'::CovMapFuncRecordReader::get<unsigned __int64,1>() - Line 751
  llvm-cov.exe!readCoverageMappingData<unsigned __int64,1>() - Line 788
  llvm-cov.exe!llvm::coverage::BinaryCoverageReader::createCoverageReaderFromBuffer() - Line 838
  llvm-cov.exe!loadBinaryFormat() - Line 986

Seems like the RequiresNullTerminator setting is one of the causes [2]. But, only a few binaries were affected by the RequiresNullTerminator and were not using mmap (VirtualAlloc) in my case when I tested with all the binaries. So, it was not a main issue for me, but I also agree that it's something we want to fix.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D99110/new/

https://reviews.llvm.org/D99110