[PATCH] D31027: [ThinLTO] Add support for emitting minimized bitcode for thin link

[Teresa Johnson via Phabricator via llvm-commits]

tejohnson created this revision.
Herald added a subscriber: Prazek.

The cumulative size of the bitcode files for a very large application
can be huge, particularly with -g. In a distributed build environment,
all of these files must be sent to the remote build node that performs
the thin link step, and this can exceed size limits.

The thin link actually only needs the summary along with a bitcode
symbol table. Until we have a proper bitcode symbol table, simply
stripping the debug metadata results in significant size reduction.

Add support for an option to additionally emit minimized bitcode
modules, just for use in the thin link step, which for now just strips
all debug metadata. I plan to add a cc1 option so this can be invoked
easily during the compile step.

However, care must be taken to ensure that these minimized thin link
bitcode files produce the same index as with the original bitcode files,
as these original bitcode files will be used in the backends.

Specifically:

1. The module hash used for caching is typically produced by hashing the

written bitcode, and we want to include the hash that would correspond
to the original bitcode file. This is because we want to ensure that
changes in the stripped portions affect caching. Added plumbing to emit
the same module hash in the minimized thin link bitcode file.

2. The module paths in the index are constructed from the module ID of

each thin linked bitcode, and typically is automatically generated from
the input file path. This is the path used for finding the modules to
import from, and obviously we need this to point to the original bitcode
files. Added support (to gold-plugin, as well as llvm-lto and llvm-lto2
for testing) to supply a suffix replacement during the thin link, that
is used to override the identifier on the MemoryBufferRef constructed
from the loaded thin link bitcode file. The assumption is that the build
system can specify that the minimized bitcode file has a name that is
similar but just as a different suffix (e.g. out.thinlink.bc instead of
out.o).

Added various tests to ensure that we get identical index files out of
the thin link step.


https://reviews.llvm.org/D31027

Files:
  include/llvm/Bitcode/BitcodeWriter.h
  include/llvm/Object/ModuleSummaryIndexObjectFile.h
  include/llvm/Transforms/IPO.h
  lib/Bitcode/Writer/BitcodeWriter.cpp
  lib/Object/ModuleSummaryIndexObjectFile.cpp
  lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
  test/ThinLTO/X86/distributed_import.ll
  test/Transforms/ThinLTOBitcodeWriter/no-type-md.ll
  test/Transforms/ThinLTOBitcodeWriter/split.ll
  test/tools/gold/X86/thinlto_object_suffix_replace.ll
  tools/gold/gold-plugin.cpp
  tools/llvm-lto/llvm-lto.cpp
  tools/llvm-lto2/llvm-lto2.cpp
  tools/opt/opt.cpp

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D31027.91985.patch
Type: text/x-patch
Size: 38448 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20170316/1460277e/attachment-0001.bin>