[PATCH] D71261: [ThinLTO] upgrade IR symtab in parallel ahead of time

[Yuanfang Chen via Phabricator via llvm-commits]

ychen created this revision.
Herald added subscribers: llvm-commits, dang, jfb, dexonsmith, steven_wu, MaskRay, hiraditya, arichardson, inglorion, mehdi_amini, mgorny, emaste.
Herald added a reviewer: espindola.
Herald added a project: LLVM.

Before parsing file, collect all bitcode files that need upgrade, and perform the actual upgrade in parallel, assuming the time taken to upgrade the symtab is proportion to the file size. Probably some of the upgrade is not necessary if the bitcode file is archive member and no symbol is defined there. After some experiments, it seems that the cost would be low because of parallelization.

Add an lld option(--cache-ir-symtab) to enable this behavior (default off). In the long run, I think it is beneficial to turn it on by default.

For symtab upgrade path, this reduces clang binary linking (all file hit lto cache) to ~5.5 (8 core, 16 threads) seconds from ~13s. Non-upgrade path takes about 3.8s.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D71261

Files:
  lld/ELF/Config.h
  lld/ELF/Driver.cpp
  lld/ELF/Driver.h
  lld/ELF/InputFiles.cpp
  lld/ELF/Options.td
  llvm/include/llvm/Bitcode/BitcodeReader.h
  llvm/include/llvm/LTO/LTO.h
  llvm/include/llvm/Object/IRObjectFile.h
  llvm/include/llvm/Object/IRSymtab.h
  llvm/include/llvm/Support/MemoryBuffer.h
  llvm/lib/LTO/LTO.cpp
  llvm/lib/Object/IRObjectFile.cpp
  llvm/lib/Object/IRSymtab.cpp
  llvm/unittests/CMakeLists.txt
  llvm/unittests/ThinLTO/CMakeLists.txt
  llvm/unittests/ThinLTO/PreUpgradeIRSymbolTable.cpp

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D71261.233061.patch
Type: text/x-patch
Size: 30213 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20191210/2caec842/attachment-0001.bin>