[llvm] r298638 - [ThinLTO] Add support for emitting minimized bitcode for thin link
Teresa Johnson via llvm-commits
llvm-commits at lists.llvm.org
Thu Mar 23 12:47:40 PDT 2017
Author: tejohnson
Date: Thu Mar 23 14:47:39 2017
New Revision: 298638
URL: http://llvm.org/viewvc/llvm-project?rev=298638&view=rev
Log:
[ThinLTO] Add support for emitting minimized bitcode for thin link
Summary:
The cumulative size of the bitcode files for a very large application
can be huge, particularly with -g. In a distributed build environment,
all of these files must be sent to the remote build node that performs
the thin link step, and this can exceed size limits.
The thin link actually only needs the summary along with a bitcode
symbol table. Until we have a proper bitcode symbol table, simply
stripping the debug metadata results in significant size reduction.
Add support for an option to additionally emit minimized bitcode
modules, just for use in the thin link step, which for now just strips
all debug metadata. I plan to add a cc1 option so this can be invoked
easily during the compile step.
However, care must be taken to ensure that these minimized thin link
bitcode files produce the same index as with the original bitcode files,
as these original bitcode files will be used in the backends.
Specifically:
1) The module hash used for caching is typically produced by hashing the
written bitcode, and we want to include the hash that would correspond
to the original bitcode file. This is because we want to ensure that
changes in the stripped portions affect caching. Added plumbing to emit
the same module hash in the minimized thin link bitcode file.
2) The module paths in the index are constructed from the module ID of
each thin linked bitcode, and typically is automatically generated from
the input file path. This is the path used for finding the modules to
import from, and obviously we need this to point to the original bitcode
files. Added gold-plugin support to take a suffix replacement during the
thin link that is used to override the identifier on the MemoryBufferRef
constructed from the loaded thin link bitcode file. The assumption is
that the build system can specify that the minimized bitcode file has a
name that is similar but uses a different suffix (e.g. out.thinlink.bc
instead of out.o).
Added various tests to ensure that we get identical index files out of
the thin link step.
Reviewers: mehdi_amini, pcc
Subscribers: Prazek, llvm-commits
Differential Revision: https://reviews.llvm.org/D31027
Added:
llvm/trunk/test/tools/gold/X86/thinlto_object_suffix_replace.ll
Modified:
llvm/trunk/include/llvm/Bitcode/BitcodeWriter.h
llvm/trunk/include/llvm/Object/ModuleSummaryIndexObjectFile.h
llvm/trunk/include/llvm/Transforms/IPO.h
llvm/trunk/lib/Bitcode/Writer/BitcodeWriter.cpp
llvm/trunk/lib/Object/ModuleSummaryIndexObjectFile.cpp
llvm/trunk/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
llvm/trunk/test/ThinLTO/X86/distributed_import.ll
llvm/trunk/test/Transforms/ThinLTOBitcodeWriter/no-type-md.ll
llvm/trunk/test/Transforms/ThinLTOBitcodeWriter/split.ll
llvm/trunk/test/Transforms/ThinLTOBitcodeWriter/unsplittable.ll
llvm/trunk/tools/gold/gold-plugin.cpp
llvm/trunk/tools/opt/opt.cpp
Modified: llvm/trunk/include/llvm/Bitcode/BitcodeWriter.h
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Bitcode/BitcodeWriter.h?rev=298638&r1=298637&r2=298638&view=diff
==============================================================================
--- llvm/trunk/include/llvm/Bitcode/BitcodeWriter.h (original)
+++ llvm/trunk/include/llvm/Bitcode/BitcodeWriter.h Thu Mar 23 14:47:39 2017
@@ -43,9 +43,16 @@ namespace llvm {
///
/// \p GenerateHash enables hashing the Module and including the hash in the
/// bitcode (currently for use in ThinLTO incremental build).
+ ///
+ /// If \p ModHash is non-null, when GenerateHash is true, the resulting
+ /// hash is written into ModHash. When GenerateHash is false, that value
+ /// is used as the hash instead of computing from the generated bitcode.
+ /// Can be used to produce the same module hash for a minimized bitcode
+ /// used just for the thin link as in the regular full bitcode that will
+ /// be used in the backend.
void writeModule(const Module *M, bool ShouldPreserveUseListOrder = false,
const ModuleSummaryIndex *Index = nullptr,
- bool GenerateHash = false);
+ bool GenerateHash = false, ModuleHash *ModHash = nullptr);
};
/// \brief Write the specified module to the specified raw output stream.
@@ -62,10 +69,18 @@ namespace llvm {
///
/// \p GenerateHash enables hashing the Module and including the hash in the
/// bitcode (currently for use in ThinLTO incremental build).
+ ///
+ /// If \p ModHash is non-null, when GenerateHash is true, the resulting
+ /// hash is written into ModHash. When GenerateHash is false, that value
+ /// is used as the hash instead of computing from the generated bitcode.
+ /// Can be used to produce the same module hash for a minimized bitcode
+ /// used just for the thin link as in the regular full bitcode that will
+ /// be used in the backend.
void WriteBitcodeToFile(const Module *M, raw_ostream &Out,
bool ShouldPreserveUseListOrder = false,
const ModuleSummaryIndex *Index = nullptr,
- bool GenerateHash = false);
+ bool GenerateHash = false,
+ ModuleHash *ModHash = nullptr);
/// Write the specified module summary index to the given raw output stream,
/// where it will be written in a new bitcode block. This is used when
Modified: llvm/trunk/include/llvm/Object/ModuleSummaryIndexObjectFile.h
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Object/ModuleSummaryIndexObjectFile.h?rev=298638&r1=298637&r2=298638&view=diff
==============================================================================
--- llvm/trunk/include/llvm/Object/ModuleSummaryIndexObjectFile.h (original)
+++ llvm/trunk/include/llvm/Object/ModuleSummaryIndexObjectFile.h Thu Mar 23 14:47:39 2017
@@ -88,9 +88,12 @@ public:
}
/// Parse the module summary index out of an IR file and return the module
-/// summary index object if found, or nullptr if not.
+/// summary index object if found, or nullptr if not. If Identifier is
+/// non-empty, it is used as the module ID (module path) in the resulting
+/// index. This can be used when the index is being read from a file
+/// containing minimized bitcode just for the thin link.
Expected<std::unique_ptr<ModuleSummaryIndex>>
-getModuleSummaryIndexForFile(StringRef Path);
+getModuleSummaryIndexForFile(StringRef Path, StringRef Identifier = "");
}
#endif
Modified: llvm/trunk/include/llvm/Transforms/IPO.h
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Transforms/IPO.h?rev=298638&r1=298637&r2=298638&view=diff
==============================================================================
--- llvm/trunk/include/llvm/Transforms/IPO.h (original)
+++ llvm/trunk/include/llvm/Transforms/IPO.h Thu Mar 23 14:47:39 2017
@@ -264,7 +264,8 @@ ModulePass *createSampleProfileLoaderPas
ModulePass *createSampleProfileLoaderPass(StringRef Name);
/// Write ThinLTO-ready bitcode to Str.
-ModulePass *createWriteThinLTOBitcodePass(raw_ostream &Str);
+ModulePass *createWriteThinLTOBitcodePass(raw_ostream &Str,
+ raw_ostream *ThinLinkOS = nullptr);
} // End llvm namespace
Modified: llvm/trunk/lib/Bitcode/Writer/BitcodeWriter.cpp
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Bitcode/Writer/BitcodeWriter.cpp?rev=298638&r1=298637&r2=298638&view=diff
==============================================================================
--- llvm/trunk/lib/Bitcode/Writer/BitcodeWriter.cpp (original)
+++ llvm/trunk/lib/Bitcode/Writer/BitcodeWriter.cpp Thu Mar 23 14:47:39 2017
@@ -108,6 +108,14 @@ class ModuleBitcodeWriter : public Bitco
/// True if a module hash record should be written.
bool GenerateHash;
+ /// If non-null, when GenerateHash is true, the resulting hash is written
+ /// into ModHash. When GenerateHash is false, that specified value
+ /// is used as the hash instead of computing from the generated bitcode.
+ /// Can be used to produce the same module hash for a minimized bitcode
+ /// used just for the thin link as in the regular full bitcode that will
+ /// be used in the backend.
+ ModuleHash *ModHash;
+
/// The start bit of the identification block.
uint64_t BitcodeStartBit;
@@ -124,10 +132,12 @@ public:
/// writing to the provided \p Buffer.
ModuleBitcodeWriter(const Module *M, SmallVectorImpl<char> &Buffer,
BitstreamWriter &Stream, bool ShouldPreserveUseListOrder,
- const ModuleSummaryIndex *Index, bool GenerateHash)
+ const ModuleSummaryIndex *Index, bool GenerateHash,
+ ModuleHash *ModHash = nullptr)
: BitcodeWriterBase(Stream), Buffer(Buffer), M(*M),
VE(*M, ShouldPreserveUseListOrder), Index(Index),
- GenerateHash(GenerateHash), BitcodeStartBit(Stream.GetCurrentBitNo()) {
+ GenerateHash(GenerateHash), ModHash(ModHash),
+ BitcodeStartBit(Stream.GetCurrentBitNo()) {
// Assign ValueIds to any callee values in the index that came from
// indirect call profiles and were recorded as a GUID not a Value*
// (which would have been assigned an ID by the ValueEnumerator).
@@ -3778,17 +3788,24 @@ static void writeIdentificationBlock(Bit
void ModuleBitcodeWriter::writeModuleHash(size_t BlockStartPos) {
// Emit the module's hash.
// MODULE_CODE_HASH: [5*i32]
- SHA1 Hasher;
- Hasher.update(ArrayRef<uint8_t>((const uint8_t *)&(Buffer)[BlockStartPos],
- Buffer.size() - BlockStartPos));
- StringRef Hash = Hasher.result();
- uint32_t Vals[5];
- for (int Pos = 0; Pos < 20; Pos += 4) {
- Vals[Pos / 4] = support::endian::read32be(Hash.data() + Pos);
- }
+ if (GenerateHash) {
+ SHA1 Hasher;
+ uint32_t Vals[5];
+ Hasher.update(ArrayRef<uint8_t>((const uint8_t *)&(Buffer)[BlockStartPos],
+ Buffer.size() - BlockStartPos));
+ StringRef Hash = Hasher.result();
+ for (int Pos = 0; Pos < 20; Pos += 4) {
+ Vals[Pos / 4] = support::endian::read32be(Hash.data() + Pos);
+ }
- // Emit the finished record.
- Stream.EmitRecord(bitc::MODULE_CODE_HASH, Vals);
+ // Emit the finished record.
+ Stream.EmitRecord(bitc::MODULE_CODE_HASH, Vals);
+
+ if (ModHash)
+ // Save the written hash value.
+ std::copy(std::begin(Vals), std::end(Vals), std::begin(*ModHash));
+ } else if (ModHash)
+ Stream.EmitRecord(bitc::MODULE_CODE_HASH, ArrayRef<uint32_t>(*ModHash));
}
void ModuleBitcodeWriter::write() {
@@ -3849,9 +3866,7 @@ void ModuleBitcodeWriter::write() {
writeValueSymbolTable(M.getValueSymbolTable(),
/* IsModuleLevel */ true, &FunctionToBitcodeIndex);
- if (GenerateHash) {
- writeModuleHash(BlockStartPos);
- }
+ writeModuleHash(BlockStartPos);
Stream.ExitBlock();
}
@@ -3942,9 +3957,10 @@ BitcodeWriter::~BitcodeWriter() = defaul
void BitcodeWriter::writeModule(const Module *M,
bool ShouldPreserveUseListOrder,
const ModuleSummaryIndex *Index,
- bool GenerateHash) {
- ModuleBitcodeWriter ModuleWriter(
- M, Buffer, *Stream, ShouldPreserveUseListOrder, Index, GenerateHash);
+ bool GenerateHash, ModuleHash *ModHash) {
+ ModuleBitcodeWriter ModuleWriter(M, Buffer, *Stream,
+ ShouldPreserveUseListOrder, Index,
+ GenerateHash, ModHash);
ModuleWriter.write();
}
@@ -3953,7 +3969,7 @@ void BitcodeWriter::writeModule(const Mo
void llvm::WriteBitcodeToFile(const Module *M, raw_ostream &Out,
bool ShouldPreserveUseListOrder,
const ModuleSummaryIndex *Index,
- bool GenerateHash) {
+ bool GenerateHash, ModuleHash *ModHash) {
SmallVector<char, 0> Buffer;
Buffer.reserve(256*1024);
@@ -3964,7 +3980,8 @@ void llvm::WriteBitcodeToFile(const Modu
Buffer.insert(Buffer.begin(), BWH_HeaderSize, 0);
BitcodeWriter Writer(Buffer);
- Writer.writeModule(M, ShouldPreserveUseListOrder, Index, GenerateHash);
+ Writer.writeModule(M, ShouldPreserveUseListOrder, Index, GenerateHash,
+ ModHash);
if (TT.isOSDarwin() || TT.isOSBinFormatMachO())
emitDarwinBCHeaderAndTrailer(Buffer, TT);
Modified: llvm/trunk/lib/Object/ModuleSummaryIndexObjectFile.cpp
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Object/ModuleSummaryIndexObjectFile.cpp?rev=298638&r1=298637&r2=298638&view=diff
==============================================================================
--- llvm/trunk/lib/Object/ModuleSummaryIndexObjectFile.cpp (original)
+++ llvm/trunk/lib/Object/ModuleSummaryIndexObjectFile.cpp Thu Mar 23 14:47:39 2017
@@ -96,13 +96,18 @@ ModuleSummaryIndexObjectFile::create(Mem
// Parse the module summary index out of an IR file and return the summary
// index object if found, or nullptr if not.
Expected<std::unique_ptr<ModuleSummaryIndex>>
-llvm::getModuleSummaryIndexForFile(StringRef Path) {
+llvm::getModuleSummaryIndexForFile(StringRef Path, StringRef Identifier) {
ErrorOr<std::unique_ptr<MemoryBuffer>> FileOrErr =
MemoryBuffer::getFileOrSTDIN(Path);
std::error_code EC = FileOrErr.getError();
if (EC)
return errorCodeToError(EC);
- MemoryBufferRef BufferRef = (FileOrErr.get())->getMemBufferRef();
+ std::unique_ptr<MemoryBuffer> MemBuffer = std::move(FileOrErr.get());
+ // If Identifier is non-empty, use it as the buffer identifier, which
+ // will become the module path in the index.
+ if (Identifier.empty())
+ Identifier = MemBuffer->getBufferIdentifier();
+ MemoryBufferRef BufferRef(MemBuffer->getBuffer(), Identifier);
if (IgnoreEmptyThinLTOIndexFile && !BufferRef.getBufferSize())
return nullptr;
Expected<std::unique_ptr<object::ModuleSummaryIndexObjectFile>> ObjOrErr =
Modified: llvm/trunk/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp?rev=298638&r1=298637&r2=298638&view=diff
==============================================================================
--- llvm/trunk/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp (original)
+++ llvm/trunk/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp Thu Mar 23 14:47:39 2017
@@ -14,7 +14,6 @@
//
//===----------------------------------------------------------------------===//
-#include "llvm/Transforms/IPO.h"
#include "llvm/Analysis/BasicAliasAnalysis.h"
#include "llvm/Analysis/ModuleSummaryAnalysis.h"
#include "llvm/Analysis/TypeMetadataUtils.h"
@@ -25,7 +24,10 @@
#include "llvm/IR/Module.h"
#include "llvm/IR/PassManager.h"
#include "llvm/Pass.h"
+#include "llvm/Support/FileSystem.h"
#include "llvm/Support/ScopedPrinter.h"
+#include "llvm/Support/raw_ostream.h"
+#include "llvm/Transforms/IPO.h"
#include "llvm/Transforms/IPO/FunctionAttrs.h"
#include "llvm/Transforms/Utils/Cloning.h"
using namespace llvm;
@@ -251,13 +253,17 @@ void forEachVirtualFunction(Constant *C,
// a multi-module bitcode file with the two parts to OS. Otherwise, write only a
// regular LTO bitcode file to OS.
void splitAndWriteThinLTOBitcode(
- raw_ostream &OS, function_ref<AAResults &(Function &)> AARGetter,
- Module &M) {
+ raw_ostream &OS, raw_ostream *ThinLinkOS,
+ function_ref<AAResults &(Function &)> AARGetter, Module &M) {
std::string ModuleId = getModuleId(&M);
if (ModuleId.empty()) {
// We couldn't generate a module ID for this module, just write it out as a
// regular LTO module.
WriteBitcodeToFile(&M, OS);
+ if (ThinLinkOS)
+ // We don't have a ThinLTO part, but still write the module to the
+ // ThinLinkOS if requested so that the expected output file is produced.
+ WriteBitcodeToFile(&M, *ThinLinkOS);
return;
}
@@ -334,17 +340,34 @@ void splitAndWriteThinLTOBitcode(
simplifyExternals(*MergedM);
- SmallVector<char, 0> Buffer;
- BitcodeWriter W(Buffer);
// FIXME: Try to re-use BSI and PFI from the original module here.
ModuleSummaryIndex Index = buildModuleSummaryIndex(M, nullptr, nullptr);
- W.writeModule(&M, /*ShouldPreserveUseListOrder=*/false, &Index,
- /*GenerateHash=*/true);
- W.writeModule(MergedM.get());
+ SmallVector<char, 0> Buffer;
+ BitcodeWriter W(Buffer);
+ // Save the module hash produced for the full bitcode, which will
+ // be used in the backends, and use that in the minimized bitcode
+ // produced for the full link.
+ ModuleHash ModHash = {{0}};
+ W.writeModule(&M, /*ShouldPreserveUseListOrder=*/false, &Index,
+ /*GenerateHash=*/true, &ModHash);
+ W.writeModule(MergedM.get());
OS << Buffer;
+
+ // If a minimized bitcode module was requested for the thin link,
+ // strip the debug info (the merged module was already stripped above)
+ // and write it to the given OS.
+ if (ThinLinkOS) {
+ Buffer.clear();
+ BitcodeWriter W2(Buffer);
+ StripDebugInfo(M);
+ W2.writeModule(&M, /*ShouldPreserveUseListOrder=*/false, &Index,
+ /*GenerateHash=*/false, &ModHash);
+ W2.writeModule(MergedM.get());
+ *ThinLinkOS << Buffer;
+ }
}
// Returns whether this module needs to be split because it uses type metadata.
@@ -359,29 +382,45 @@ bool requiresSplit(Module &M) {
return false;
}
-void writeThinLTOBitcode(raw_ostream &OS,
+void writeThinLTOBitcode(raw_ostream &OS, raw_ostream *ThinLinkOS,
function_ref<AAResults &(Function &)> AARGetter,
Module &M, const ModuleSummaryIndex *Index) {
// See if this module has any type metadata. If so, we need to split it.
if (requiresSplit(M))
- return splitAndWriteThinLTOBitcode(OS, AARGetter, M);
+ return splitAndWriteThinLTOBitcode(OS, ThinLinkOS, AARGetter, M);
// Otherwise we can just write it out as a regular module.
+
+ // Save the module hash produced for the full bitcode, which will
+ // be used in the backends, and use that in the minimized bitcode
+ // produced for the full link.
+ ModuleHash ModHash = {{0}};
WriteBitcodeToFile(&M, OS, /*ShouldPreserveUseListOrder=*/false, Index,
- /*GenerateHash=*/true);
+ /*GenerateHash=*/true, &ModHash);
+ // If a minimized bitcode module was requested for the thin link,
+ // strip the debug info and write it to the given OS.
+ if (ThinLinkOS) {
+ StripDebugInfo(M);
+ WriteBitcodeToFile(&M, *ThinLinkOS, /*ShouldPreserveUseListOrder=*/false,
+ Index,
+ /*GenerateHash=*/false, &ModHash);
+ }
}
class WriteThinLTOBitcode : public ModulePass {
raw_ostream &OS; // raw_ostream to print on
+ // The output stream on which to emit a minimized module for use
+ // just in the thin link, if requested.
+ raw_ostream *ThinLinkOS;
public:
static char ID; // Pass identification, replacement for typeid
- WriteThinLTOBitcode() : ModulePass(ID), OS(dbgs()) {
+ WriteThinLTOBitcode() : ModulePass(ID), OS(dbgs()), ThinLinkOS(nullptr) {
initializeWriteThinLTOBitcodePass(*PassRegistry::getPassRegistry());
}
- explicit WriteThinLTOBitcode(raw_ostream &o)
- : ModulePass(ID), OS(o) {
+ explicit WriteThinLTOBitcode(raw_ostream &o, raw_ostream *ThinLinkOS)
+ : ModulePass(ID), OS(o), ThinLinkOS(ThinLinkOS) {
initializeWriteThinLTOBitcodePass(*PassRegistry::getPassRegistry());
}
@@ -390,7 +429,7 @@ public:
bool runOnModule(Module &M) override {
const ModuleSummaryIndex *Index =
&(getAnalysis<ModuleSummaryIndexWrapperPass>().getIndex());
- writeThinLTOBitcode(OS, LegacyAARGetter(*this), M, Index);
+ writeThinLTOBitcode(OS, ThinLinkOS, LegacyAARGetter(*this), M, Index);
return true;
}
void getAnalysisUsage(AnalysisUsage &AU) const override {
@@ -411,6 +450,7 @@ INITIALIZE_PASS_DEPENDENCY(TargetLibrary
INITIALIZE_PASS_END(WriteThinLTOBitcode, "write-thinlto-bitcode",
"Write ThinLTO Bitcode", false, true)
-ModulePass *llvm::createWriteThinLTOBitcodePass(raw_ostream &Str) {
- return new WriteThinLTOBitcode(Str);
+ModulePass *llvm::createWriteThinLTOBitcodePass(raw_ostream &Str,
+ raw_ostream *ThinLinkOS) {
+ return new WriteThinLTOBitcode(Str, ThinLinkOS);
}
Modified: llvm/trunk/test/ThinLTO/X86/distributed_import.ll
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/ThinLTO/X86/distributed_import.ll?rev=298638&r1=298637&r2=298638&view=diff
==============================================================================
--- llvm/trunk/test/ThinLTO/X86/distributed_import.ll (original)
+++ llvm/trunk/test/ThinLTO/X86/distributed_import.ll Thu Mar 23 14:47:39 2017
@@ -1,15 +1,50 @@
-; RUN: opt -module-summary %s -o %t1.bc
-; RUN: opt -module-summary %p/Inputs/distributed_import.ll -o %t2.bc
+; Test distributed build thin link output from llvm-lto2
+; Generate bitcode files with summary, as well as minimized bitcode without
+; the debug metadata for the thin link.
+; RUN: opt -thinlto-bc %s -thin-link-bitcode-file=%t1.thinlink.bc -o %t1.bc
+; RUN: opt -thinlto-bc %p/Inputs/distributed_import.ll -thin-link-bitcode-file=%t2.thinlink.bc -o %t2.bc
+
+; First perform the thin link on the normal bitcode file.
; RUN: llvm-lto2 %t1.bc %t2.bc -o %t.o -save-temps \
; RUN: -thinlto-distributed-indexes \
; RUN: -r=%t1.bc,g, \
; RUN: -r=%t1.bc,f,px \
; RUN: -r=%t2.bc,g,px
-; RUN: opt -function-import -summary-file %t1.bc.thinlto.bc %t1.bc -o %t1.out
+; RUN: opt -function-import -summary-file %t1.bc.thinlto.bc %t1.bc -o %t1.out
; RUN: opt -function-import -summary-file %t2.bc.thinlto.bc %t2.bc -o %t2.out
; RUN: llvm-dis -o - %t2.out | FileCheck %s
-; CHECK: @G.llvm.0
+
+; Save the generated index files.
+; RUN: cp %t1.bc.thinlto.bc %t1.bc.thinlto.bc.orig
+; RUN: cp %t2.bc.thinlto.bc %t2.bc.thinlto.bc.orig
+
+; Copy the minimized bitcode to the regular bitcode path so the module
+; paths in the index are the same (save the regular bitcode for use again
+; further down).
+; RUN: cp %t1.bc %t1.bc.sv
+; RUN: cp %t1.thinlink.bc %t1.bc
+; RUN: cp %t2.bc %t2.bc.sv
+; RUN: cp %t2.thinlink.bc %t2.bc
+
+; Next perform the thin link on the minimized bitcode files, and compare dumps
+; of the resulting indexes to the above dumps to ensure they are identical.
+; RUN: rm -f %t1.bc.thinlto.bc %t2.bc.thinlto.bc
+; RUN: llvm-lto2 %t1.bc %t2.bc -o %t.o -save-temps \
+; RUN: -thinlto-distributed-indexes \
+; RUN: -r=%t1.bc,g, \
+; RUN: -r=%t1.bc,f,px \
+; RUN: -r=%t2.bc,g,px
+; RUN: diff %t1.bc.thinlto.bc.orig %t1.bc.thinlto.bc
+; RUN: diff %t2.bc.thinlto.bc.orig %t2.bc.thinlto.bc
+
+; Make sure importing occurs as expected
+; RUN: cp %t1.bc.sv %t1.bc
+; RUN: cp %t2.bc.sv %t2.bc
+; RUN: opt -function-import -summary-file %t2.bc.thinlto.bc %t2.bc -o %t2.out
+; RUN: llvm-dis -o - %t2.out | FileCheck %s
+
+; CHECK: @G.llvm.
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
@@ -20,3 +55,8 @@ entry:
call i32 (...) @g()
ret void
}
+
+!llvm.dbg.cu = !{}
+
+!1 = !{i32 2, !"Debug Info Version", i32 3}
+!llvm.module.flags = !{!1}
Modified: llvm/trunk/test/Transforms/ThinLTOBitcodeWriter/no-type-md.ll
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/ThinLTOBitcodeWriter/no-type-md.ll?rev=298638&r1=298637&r2=298638&view=diff
==============================================================================
--- llvm/trunk/test/Transforms/ThinLTOBitcodeWriter/no-type-md.ll (original)
+++ llvm/trunk/test/Transforms/ThinLTOBitcodeWriter/no-type-md.ll Thu Mar 23 14:47:39 2017
@@ -1,6 +1,30 @@
-; RUN: opt -thinlto-bc -o %t %s
-; RUN: llvm-dis -o - %t | FileCheck %s
-; RUN: llvm-bcanalyzer -dump %t | FileCheck --check-prefix=BCA %s
+; Generate bitcode files with summary, as well as minimized bitcode without
+; the debug metadata for the thin link.
+; RUN: opt -thinlto-bc -thin-link-bitcode-file=%t.thinlink.bc -o %t.bc %s
+; RUN: llvm-dis -o - %t.bc | FileCheck %s
+; RUN: llvm-dis -o - %t.thinlink.bc | FileCheck --check-prefix=NODEBUG %s
+; RUN: llvm-bcanalyzer -dump %t.bc | FileCheck --check-prefix=BCA %s
+
+; Make sure the combined index files produced by both the normal and the
+; thin link bitcode files are identical
+; RUN: llvm-lto -thinlto -o %t3 %t.bc
+; Copy the minimized bitcode to the regular bitcode path so the module
+; paths in the index are the same (save and restore the regular bitcode
+; for use again further down).
+; RUN: mv %t.bc %t.bc.sv
+; RUN: cp %t.thinlink.bc %t.bc
+; RUN: llvm-lto -thinlto -o %t4 %t.bc
+; RUN: mv %t.bc.sv %t.bc
+; RUN: diff %t3.thinlto.bc %t4.thinlto.bc
+
+; Try again using -thinlto-action to produce combined index
+; RUN: rm -f %t3.thinlto.bc %t4.thinlto.bc
+; RUN: llvm-lto -thinlto-action=thinlink -o %t3.thinlto.bc %t.bc
+; Copy the minimized bitcode to the regular bitcode path so the module
+; paths in the index are the same.
+; RUN: cp %t.thinlink.bc %t.bc
+; RUN: llvm-lto -thinlto-action=thinlink -o %t4.thinlto.bc %t.bc
+; RUN: diff %t3.thinlto.bc %t4.thinlto.bc
; BCA: <GLOBALVAL_SUMMARY_BLOCK
@@ -11,3 +35,10 @@
define void @f() {
ret void
}
+
+; CHECK: !llvm.dbg.cu
+; NODEBUG-NOT: !llvm.dbg.cu
+!llvm.dbg.cu = !{}
+
+!1 = !{i32 2, !"Debug Info Version", i32 3}
+!llvm.module.flags = !{!1}
Modified: llvm/trunk/test/Transforms/ThinLTOBitcodeWriter/split.ll
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/ThinLTOBitcodeWriter/split.ll?rev=298638&r1=298637&r2=298638&view=diff
==============================================================================
--- llvm/trunk/test/Transforms/ThinLTOBitcodeWriter/split.ll (original)
+++ llvm/trunk/test/Transforms/ThinLTOBitcodeWriter/split.ll Thu Mar 23 14:47:39 2017
@@ -1,11 +1,26 @@
-; RUN: opt -thinlto-bc -o %t %s
-; RUN: llvm-modextract -b -n 0 -o %t0 %t
-; RUN: llvm-modextract -b -n 1 -o %t1 %t
+; Generate bitcode files with summary, as well as minimized bitcode without
+; the debug metadata for the thin link.
+; RUN: opt -thinlto-bc -thin-link-bitcode-file=%t2 -o %t %s
+; RUN: llvm-modextract -b -n 0 -o %t0.bc %t
+; RUN: llvm-modextract -b -n 1 -o %t1.bc %t
+; RUN: llvm-modextract -b -n 0 -o %t0.thinlink.bc %t2
+; RUN: llvm-modextract -b -n 1 -o %t1.thinlink.bc %t2
; RUN: not llvm-modextract -b -n 2 -o - %t 2>&1 | FileCheck --check-prefix=ERROR %s
-; RUN: llvm-dis -o - %t0 | FileCheck --check-prefix=M0 %s
-; RUN: llvm-dis -o - %t1 | FileCheck --check-prefix=M1 %s
-; RUN: llvm-bcanalyzer -dump %t0 | FileCheck --check-prefix=BCA0 %s
-; RUN: llvm-bcanalyzer -dump %t1 | FileCheck --check-prefix=BCA1 %s
+; RUN: llvm-dis -o - %t0.bc | FileCheck --check-prefix=M0 %s
+; RUN: llvm-dis -o - %t1.bc | FileCheck --check-prefix=M1 %s
+; RUN: llvm-dis -o - %t0.thinlink.bc | FileCheck --check-prefix=NODEBUG %s
+; RUN: llvm-dis -o - %t1.thinlink.bc | FileCheck --check-prefix=NODEBUG %s
+; RUN: llvm-bcanalyzer -dump %t0.bc | FileCheck --check-prefix=BCA0 %s
+; RUN: llvm-bcanalyzer -dump %t1.bc | FileCheck --check-prefix=BCA1 %s
+
+; Make sure the combined index files produced by both the normal and the
+; thin link bitcode files are identical
+; RUN: llvm-lto -thinlto -o %t3 %t0.bc
+; Copy the minimized bitcode to the regular bitcode path so the module
+; paths in the index are the same.
+; RUN: cp %t0.thinlink.bc %t0.bc
+; RUN: llvm-lto -thinlto -o %t4 %t0.bc
+; RUN: diff %t3.thinlto.bc %t4.thinlto.bc
; ERROR: llvm-modextract: error: module index out of range; bitcode file contains 2 module(s)
@@ -29,6 +44,7 @@ define i8* @f() {
; M0: !llvm.dbg.cu
; M1-NOT: !llvm.dbg.cu
+; NODEBUG-NOT: !llvm.dbg.cu
!llvm.dbg.cu = !{}
!1 = !{i32 2, !"Debug Info Version", i32 3}
Modified: llvm/trunk/test/Transforms/ThinLTOBitcodeWriter/unsplittable.ll
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/ThinLTOBitcodeWriter/unsplittable.ll?rev=298638&r1=298637&r2=298638&view=diff
==============================================================================
--- llvm/trunk/test/Transforms/ThinLTOBitcodeWriter/unsplittable.ll (original)
+++ llvm/trunk/test/Transforms/ThinLTOBitcodeWriter/unsplittable.ll Thu Mar 23 14:47:39 2017
@@ -1,6 +1,9 @@
-; RUN: opt -thinlto-bc -o %t %s
+; RUN: opt -thinlto-bc -thin-link-bitcode-file=%t2 -o %t %s
; RUN: llvm-dis -o - %t | FileCheck %s
; RUN: llvm-bcanalyzer -dump %t | FileCheck --check-prefix=BCA %s
+; When not splitting the module, the thin link bitcode file should simply be a
+; copy of the regular module.
+; RUN: diff %t %t2
; BCA-NOT: <GLOBALVAL_SUMMARY_BLOCK
Added: llvm/trunk/test/tools/gold/X86/thinlto_object_suffix_replace.ll
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/tools/gold/X86/thinlto_object_suffix_replace.ll?rev=298638&view=auto
==============================================================================
--- llvm/trunk/test/tools/gold/X86/thinlto_object_suffix_replace.ll (added)
+++ llvm/trunk/test/tools/gold/X86/thinlto_object_suffix_replace.ll Thu Mar 23 14:47:39 2017
@@ -0,0 +1,41 @@
+; Test to make sure the thinlto-object-suffix-replace option is handled
+; correctly.
+
+; Generate bitcode file with summary, as well as a minimized bitcode without
+; the debug metadata for the thin link.
+; RUN: opt -thinlto-bc %s -thin-link-bitcode-file=%t1.thinlink.bc -o %t1.o
+
+; First perform the thin link on the normal bitcode file, and save the
+; resulting index.
+; RUN: %gold -plugin %llvmshlibdir/LLVMgold.so \
+; RUN: -m elf_x86_64 \
+; RUN: --plugin-opt=thinlto \
+; RUN: --plugin-opt=thinlto-index-only \
+; RUN: -shared %t1.o -o %t3
+; RUN: cp %t1.o.thinlto.bc %t1.o.thinlto.bc.orig
+
+; Next perform the thin link on the minimized bitcode file, and compare dump
+; of the resulting index to the above dump to ensure they are identical.
+; RUN: rm -f %t1.o.thinlto.bc
+; Make sure it isn't inadvertently using the regular bitcode file.
+; RUN: rm -f %t1.o
+; RUN: %gold -plugin %llvmshlibdir/LLVMgold.so \
+; RUN: -m elf_x86_64 \
+; RUN: --plugin-opt=thinlto \
+; RUN: --plugin-opt=thinlto-index-only \
+; RUN: --plugin-opt=thinlto-object-suffix-replace=".thinlink.bc;.o" \
+; RUN: -shared %t1.thinlink.bc -o %t3
+; RUN: diff %t1.o.thinlto.bc.orig %t1.o.thinlto.bc
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @f() {
+entry:
+ ret void
+}
+
+!llvm.dbg.cu = !{}
+
+!1 = !{i32 2, !"Debug Info Version", i32 3}
+!llvm.module.flags = !{!1}
Modified: llvm/trunk/tools/gold/gold-plugin.cpp
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/tools/gold/gold-plugin.cpp?rev=298638&r1=298637&r2=298638&view=diff
==============================================================================
--- llvm/trunk/tools/gold/gold-plugin.cpp (original)
+++ llvm/trunk/tools/gold/gold-plugin.cpp Thu Mar 23 14:47:39 2017
@@ -164,6 +164,12 @@ namespace options {
// corresponding bitcode file, will use a path formed by replacing the
// bitcode file's path prefix matching oldprefix with newprefix.
static std::string thinlto_prefix_replace;
+ // Option to control the name of modules encoded in the individual index
+ // files for a distributed backend. This enables the use of minimized
+ // bitcode files for the thin link, assuming the name of the full bitcode
+ // file used in the backend differs just in some part of the file suffix.
+ // If specified, expects a string of the form "oldsuffix:newsuffix".
+ static std::string thinlto_object_suffix_replace;
// Optional path to a directory for caching ThinLTO objects.
static std::string cache_dir;
// Additional options to pass into the code generator.
@@ -206,6 +212,12 @@ namespace options {
thinlto_prefix_replace = opt.substr(strlen("thinlto-prefix-replace="));
if (thinlto_prefix_replace.find(';') == std::string::npos)
message(LDPL_FATAL, "thinlto-prefix-replace expects 'old;new' format");
+ } else if (opt.startswith("thinlto-object-suffix-replace=")) {
+ thinlto_object_suffix_replace =
+ opt.substr(strlen("thinlto-object-suffix-replace="));
+ if (thinlto_object_suffix_replace.find(';') == std::string::npos)
+ message(LDPL_FATAL,
+ "thinlto-object-suffix-replace expects 'old;new' format");
} else if (opt.startswith("cache-dir=")) {
cache_dir = opt.substr(strlen("cache-dir="));
} else if (opt.size() == 2 && opt[0] == 'O') {
@@ -566,8 +578,35 @@ static const void *getSymbolsAndView(cla
return View;
}
-static void addModule(LTO &Lto, claimed_file &F, const void *View) {
- MemoryBufferRef BufferRef(StringRef((const char *)View, F.filesize), F.name);
+/// Parse the thinlto-object-suffix-replace option into the \p OldSuffix and
+/// \p NewSuffix strings, if it was specified.
+static void getThinLTOOldAndNewSuffix(std::string &OldSuffix,
+ std::string &NewSuffix) {
+ assert(options::thinlto_object_suffix_replace.empty() ||
+ options::thinlto_object_suffix_replace.find(";") != StringRef::npos);
+ StringRef SuffixReplace = options::thinlto_object_suffix_replace;
+ std::pair<StringRef, StringRef> Split = SuffixReplace.split(";");
+ OldSuffix = Split.first.str();
+ NewSuffix = Split.second.str();
+}
+
+/// Given the original \p Path to an output file, replace any filename
+/// suffix matching \p OldSuffix with \p NewSuffix.
+static std::string getThinLTOObjectFileName(const std::string Path,
+ const std::string &OldSuffix,
+ const std::string &NewSuffix) {
+ if (OldSuffix.empty() && NewSuffix.empty())
+ return Path;
+ StringRef NewPath = Path;
+ NewPath.consume_back(OldSuffix);
+ std::string NewNewPath = NewPath.str() + NewSuffix;
+ return NewPath.str() + NewSuffix;
+}
+
+static void addModule(LTO &Lto, claimed_file &F, const void *View,
+ StringRef Filename) {
+ MemoryBufferRef BufferRef(StringRef((const char *)View, F.filesize),
+ Filename);
Expected<std::unique_ptr<InputFile>> ObjOrErr = InputFile::create(BufferRef);
if (!ObjOrErr)
@@ -789,19 +828,31 @@ static ld_plugin_status allSymbolsReadHo
if (options::thinlto_index_only)
getThinLTOOldAndNewPrefix(OldPrefix, NewPrefix);
+ std::string OldSuffix, NewSuffix;
+ getThinLTOOldAndNewSuffix(OldSuffix, NewSuffix);
+ // Set for owning string objects used as buffer identifiers.
+ StringSet<> ObjectFilenames;
+
for (claimed_file &F : Modules) {
if (options::thinlto && !HandleToInputFile.count(F.leader_handle))
HandleToInputFile.insert(std::make_pair(
F.leader_handle, llvm::make_unique<PluginInputFile>(F.handle)));
const void *View = getSymbolsAndView(F);
+ // In case we are thin linking with a minimized bitcode file, ensure
+ // the module paths encoded in the index reflect where the backends
+ // will locate the full bitcode files for compiling/importing.
+ std::string Identifier =
+ getThinLTOObjectFileName(F.name, OldSuffix, NewSuffix);
+ auto ObjFilename = ObjectFilenames.insert(Identifier);
+ assert(ObjFilename.second);
if (!View) {
if (options::thinlto_index_only)
// Write empty output files that may be expected by the distributed
// build system.
- writeEmptyDistributedBuildOutputs(F.name, OldPrefix, NewPrefix);
+ writeEmptyDistributedBuildOutputs(Identifier, OldPrefix, NewPrefix);
continue;
}
- addModule(*Lto, F, View);
+ addModule(*Lto, F, View, ObjFilename.first->first());
}
SmallString<128> Filename;
Modified: llvm/trunk/tools/opt/opt.cpp
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/tools/opt/opt.cpp?rev=298638&r1=298637&r2=298638&view=diff
==============================================================================
--- llvm/trunk/tools/opt/opt.cpp (original)
+++ llvm/trunk/tools/opt/opt.cpp Thu Mar 23 14:47:39 2017
@@ -102,6 +102,11 @@ static cl::opt<bool>
OutputThinLTOBC("thinlto-bc",
cl::desc("Write output as ThinLTO-ready bitcode"));
+static cl::opt<std::string> ThinLinkBitcodeFile(
+ "thin-link-bitcode-file", cl::value_desc("filename"),
+ cl::desc(
+ "A file in which to write minimized bitcode for the thin link only"));
+
static cl::opt<bool>
NoVerify("disable-verify", cl::desc("Do not run the verifier"), cl::Hidden);
@@ -456,6 +461,7 @@ int main(int argc, char **argv) {
// Figure out what stream we are supposed to write to...
std::unique_ptr<tool_output_file> Out;
+ std::unique_ptr<tool_output_file> ThinLinkOut;
if (NoOutput) {
if (!OutputFilename.empty())
errs() << "WARNING: The -o (output filename) option is ignored when\n"
@@ -471,6 +477,15 @@ int main(int argc, char **argv) {
errs() << EC.message() << '\n';
return 1;
}
+
+ if (!ThinLinkBitcodeFile.empty()) {
+ ThinLinkOut.reset(
+ new tool_output_file(ThinLinkBitcodeFile, EC, sys::fs::F_None));
+ if (EC) {
+ errs() << EC.message() << '\n';
+ return 1;
+ }
+ }
}
Triple ModuleTriple(M->getTargetTriple());
@@ -700,7 +715,8 @@ int main(int argc, char **argv) {
report_fatal_error("Text output is incompatible with -module-hash");
Passes.add(createPrintModulePass(*OS, "", PreserveAssemblyUseListOrder));
} else if (OutputThinLTOBC)
- Passes.add(createWriteThinLTOBitcodePass(*OS));
+ Passes.add(createWriteThinLTOBitcodePass(
+ *OS, ThinLinkOut ? &ThinLinkOut->os() : nullptr));
else
Passes.add(createBitcodeWriterPass(*OS, PreserveBitcodeUseListOrder,
EmitSummaryIndex, EmitModuleHash));
@@ -747,5 +763,8 @@ int main(int argc, char **argv) {
if (YamlFile)
YamlFile->keep();
+ if (ThinLinkOut)
+ ThinLinkOut->keep();
+
return 0;
}
More information about the llvm-commits
mailing list