[lld] r299168 - LTO: Reduce memory consumption by creating an in-memory symbol table for InputFiles. NFCI.

[Peter Collingbourne via llvm-commits]

Author: pcc
Date: Thu Mar 30 21:28:30 2017
New Revision: 299168

URL: http://llvm.org/viewvc/llvm-project?rev=299168&view=rev
Log:
LTO: Reduce memory consumption by creating an in-memory symbol table for InputFiles. NFCI.

Introduce symbol table data structures that can be potentially written to
disk, have the LTO library build those data structures using temporarily
constructed modules and redirect the LTO library implementation to go through
those data structures. This allows us to remove the LLVMContext and Modules
owned by InputFile.

With this change I measured a peak memory consumption decrease from 5.4GB to
2.8GB in a no-op incremental ThinLTO link of Chromium on Linux. The impact on
memory consumption is larger in COFF linkers where we are currently forced
to materialize all metadata in order to read linker options. Peak memory
consumption linking a large piece of Chromium for Windows with full LTO and
debug info decreases from >64GB (OOM) to 15GB.

Part of PR27551.

Differential Revision: https://reviews.llvm.org/D31364

Modified:
    lld/trunk/COFF/InputFiles.cpp
    lld/trunk/ELF/InputFiles.cpp

Modified: lld/trunk/COFF/InputFiles.cpp
URL: http://llvm.org/viewvc/llvm-project/lld/trunk/COFF/InputFiles.cpp?rev=299168&r1=299167&r2=299168&view=diff
==============================================================================
--- lld/trunk/COFF/InputFiles.cpp (original)
+++ lld/trunk/COFF/InputFiles.cpp Thu Mar 30 21:28:30 2017
@@ -355,13 +355,12 @@ void BitcodeFile::parse() {
       SymbolBody *Alias = Symtab->addUndefined(Saver.save(Fallback));
       checkAndSetWeakAlias(Symtab, this, Sym->body(), Alias);
     } else {
-      Expected<int> ComdatIndex = ObjSym.getComdatIndex();
-      bool IsCOMDAT = ComdatIndex && *ComdatIndex != -1;
+      bool IsCOMDAT = ObjSym.getComdatIndex() != -1;
       Sym = Symtab->addRegular(this, SymName, IsCOMDAT);
     }
     SymbolBodies.push_back(Sym->body());
   }
-  Directives = check(Obj->getLinkerOpts());
+  Directives = Obj->getCOFFLinkerOpts();
 }
 
 MachineTypes BitcodeFile::getMachineType() {

Modified: lld/trunk/ELF/InputFiles.cpp
URL: http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/InputFiles.cpp?rev=299168&r1=299167&r2=299168&view=diff
==============================================================================
--- lld/trunk/ELF/InputFiles.cpp (original)
+++ lld/trunk/ELF/InputFiles.cpp Thu Mar 30 21:28:30 2017
@@ -818,7 +818,7 @@ static Symbol *createBitcodeSymbol(const
   uint8_t Visibility = mapVisibility(ObjSym.getVisibility());
   bool CanOmitFromDynSym = ObjSym.canBeOmittedFromSymbolTable();
 
-  int C = check(ObjSym.getComdatIndex(), F->LogName);
+  int C = ObjSym.getComdatIndex();
   if (C != -1 && !KeptComdats[C])
     return Symtab<ELFT>::X->addUndefined(NameRef, /*IsLocal=*/false, Binding,
                                          Visibility, Type, CanOmitFromDynSym,
@@ -855,10 +855,8 @@ void BitcodeFile::parse(DenseSet<CachedH
   Obj = check(lto::InputFile::create(MBRef), this->LogName);
 
   std::vector<bool> KeptComdats;
-  for (StringRef S : Obj->getComdatTable()) {
-    StringRef N = Saver.save(S);
-    KeptComdats.push_back(ComdatGroups.insert(CachedHashStringRef(N)).second);
-  }
+  for (StringRef S : Obj->getComdatTable())
+    KeptComdats.push_back(ComdatGroups.insert(CachedHashStringRef(S)).second);
 
   for (const lto::InputFile::Symbol &ObjSym : Obj->symbols())
     Symbols.push_back(createBitcodeSymbol<ELFT>(KeptComdats, ObjSym, this));