[PATCH] D31364: LTO: Reduce memory consumption by creating an in-memory symbol table for InputFiles. NFCI.

[Peter Collingbourne via Phabricator via llvm-commits]

pcc created this revision.
Herald added subscribers: aprantl, mgorny.

Introduce symbol table data structures that can be potentially written to
disk, have the LTO library build those data structures using temporarily
constructed modules and redirect the LTO library implementation to go through
those data structures. This allows us to remove the LLVMContext and Modules
owned by InputFile.

With this change I measured a peak memory consumption decrease from 5.4GB to
2.8GB in a no-op incremental ThinLTO link of Chromium on Linux, and total
time elapsed decreases from ~61s to ~48s. The impact on memory consumption
is larger in COFF linkers where we are currently forced to materialize all
metadata in order to read linker options. Peak memory consumption linking a
large piece of Chromium for Windows with full LTO and debug info decreases
from >64GB (OOM) to 15GB.

Part of PR27551.


https://reviews.llvm.org/D31364

Files:
  lld/COFF/InputFiles.cpp
  lld/COFF/LTO.cpp
  lld/ELF/InputFiles.cpp
  lld/ELF/LTO.cpp
  llvm/include/llvm/LTO/LTO.h
  llvm/include/llvm/Object/IRSymtab.h
  llvm/lib/LTO/LTO.cpp
  llvm/lib/LTO/LTOBackend.cpp
  llvm/lib/Object/CMakeLists.txt
  llvm/lib/Object/IRSymtab.cpp
  llvm/tools/gold/gold-plugin.cpp
  llvm/tools/llvm-lto2/llvm-lto2.cpp

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D31364.93032.patch
Type: text/x-patch
Size: 43465 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20170325/4283ed75/attachment.bin>