r330194 - [ThinLTO] Pass -save-temps to LTO backend for distributed ThinLTO builds
Teresa Johnson via cfe-commits
cfe-commits at lists.llvm.org
Tue Apr 17 09:39:25 PDT 2018
Author: tejohnson
Date: Tue Apr 17 09:39:25 2018
New Revision: 330194
URL: http://llvm.org/viewvc/llvm-project?rev=330194&view=rev
Log:
[ThinLTO] Pass -save-temps to LTO backend for distributed ThinLTO builds
Summary:
The clang driver option -save-temps was not passed to the LTO config,
so when invoking the ThinLTO backends via clang during distributed
builds there was no way to get LTO to save temp files.
Getting this to work with ThinLTO distributed builds also required
changing the driver to avoid a separate compile step to emit unoptimized
bitcode when the input was already bitcode under -save-temps. Not only is
this unnecessary in general, it is problematic for ThinLTO backends since
the temporary bitcode file to the backend would not match the module path
in the combined index, leading to incorrect ThinLTO backend index-based
optimizations.
Reviewers: pcc
Subscribers: mehdi_amini, inglorion, eraman, cfe-commits
Differential Revision: https://reviews.llvm.org/D45217
Modified:
cfe/trunk/include/clang/Driver/Options.td
cfe/trunk/include/clang/Frontend/CodeGenOptions.h
cfe/trunk/lib/CodeGen/BackendUtil.cpp
cfe/trunk/lib/Driver/Driver.cpp
cfe/trunk/lib/Driver/ToolChains/Clang.cpp
cfe/trunk/lib/Frontend/CompilerInvocation.cpp
cfe/trunk/test/CodeGen/thinlto_backend.ll
cfe/trunk/test/Driver/thinlto_backend.c
Modified: cfe/trunk/include/clang/Driver/Options.td
URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Driver/Options.td?rev=330194&r1=330193&r2=330194&view=diff
==============================================================================
--- cfe/trunk/include/clang/Driver/Options.td (original)
+++ cfe/trunk/include/clang/Driver/Options.td Tue Apr 17 09:39:25 2018
@@ -2275,7 +2275,7 @@ def fno_rtlib_add_rpath: Flag<["-"], "fn
HelpText<"Do not add -rpath with architecture-specific resource directory to the linker flags">;
def r : Flag<["-"], "r">, Flags<[LinkerInput,NoArgumentUnused]>,
Group<Link_Group>;
-def save_temps_EQ : Joined<["-", "--"], "save-temps=">, Flags<[DriverOption]>,
+def save_temps_EQ : Joined<["-", "--"], "save-temps=">, Flags<[CC1Option, DriverOption]>,
HelpText<"Save intermediate compilation results.">;
def save_temps : Flag<["-", "--"], "save-temps">, Flags<[DriverOption]>,
Alias<save_temps_EQ>, AliasArgs<["cwd"]>,
Modified: cfe/trunk/include/clang/Frontend/CodeGenOptions.h
URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Frontend/CodeGenOptions.h?rev=330194&r1=330193&r2=330194&view=diff
==============================================================================
--- cfe/trunk/include/clang/Frontend/CodeGenOptions.h (original)
+++ cfe/trunk/include/clang/Frontend/CodeGenOptions.h Tue Apr 17 09:39:25 2018
@@ -203,6 +203,9 @@ public:
/// the summary and module symbol table (and not, e.g. any debug metadata).
std::string ThinLinkBitcodeFile;
+ /// Prefix to use for -save-temps output.
+ std::string SaveTempsFilePrefix;
+
/// Name of file passed with -fcuda-include-gpubinary option to forward to
/// CUDA runtime back-end for incorporating them into host-side object file.
std::string CudaGpuBinaryFileName;
Modified: cfe/trunk/lib/CodeGen/BackendUtil.cpp
URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/CodeGen/BackendUtil.cpp?rev=330194&r1=330193&r2=330194&view=diff
==============================================================================
--- cfe/trunk/lib/CodeGen/BackendUtil.cpp (original)
+++ cfe/trunk/lib/CodeGen/BackendUtil.cpp Tue Apr 17 09:39:25 2018
@@ -1117,6 +1117,15 @@ static void runThinLTOBackend(ModuleSumm
return llvm::make_unique<lto::NativeObjectStream>(std::move(OS));
};
lto::Config Conf;
+ if (CGOpts.SaveTempsFilePrefix != "") {
+ if (Error E = Conf.addSaveTemps(CGOpts.SaveTempsFilePrefix + ".",
+ /* UseInputModulePath */ false)) {
+ handleAllErrors(std::move(E), [&](ErrorInfoBase &EIB) {
+ errs() << "Error setting up ThinLTO save-temps: " << EIB.message()
+ << '\n';
+ });
+ }
+ }
Conf.CPU = TOpts.CPU;
Conf.CodeModel = getCodeModel(CGOpts);
Conf.MAttrs = TOpts.Features;
Modified: cfe/trunk/lib/Driver/Driver.cpp
URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Driver/Driver.cpp?rev=330194&r1=330193&r2=330194&view=diff
==============================================================================
--- cfe/trunk/lib/Driver/Driver.cpp (original)
+++ cfe/trunk/lib/Driver/Driver.cpp Tue Apr 17 09:39:25 2018
@@ -3379,19 +3379,34 @@ class ToolSelector final {
const Tool *combineBackendCompile(ArrayRef<JobActionInfo> ActionInfo,
const ActionList *&Inputs,
ActionList &CollapsedOffloadAction) {
- if (ActionInfo.size() < 2 || !canCollapsePreprocessorAction())
+ if (ActionInfo.size() < 2)
return nullptr;
auto *BJ = dyn_cast<BackendJobAction>(ActionInfo[0].JA);
auto *CJ = dyn_cast<CompileJobAction>(ActionInfo[1].JA);
if (!BJ || !CJ)
return nullptr;
+ // Check if the initial input (to the compile job or its predessor if one
+ // exists) is LLVM bitcode. In that case, no preprocessor step is required
+ // and we can still collapse the compile and backend jobs when we have
+ // -save-temps. I.e. there is no need for a separate compile job just to
+ // emit unoptimized bitcode.
+ bool InputIsBitcode = true;
+ for (size_t i = 1; i < ActionInfo.size(); i++)
+ if (ActionInfo[i].JA->getType() != types::TY_LLVM_BC &&
+ ActionInfo[i].JA->getType() != types::TY_LTO_BC) {
+ InputIsBitcode = false;
+ break;
+ }
+ if (!InputIsBitcode && !canCollapsePreprocessorAction())
+ return nullptr;
+
// Get compiler tool.
const Tool *T = TC.SelectTool(*CJ);
if (!T)
return nullptr;
- if (T->canEmitIR() && (SaveTemps || EmbedBitcode))
+ if (T->canEmitIR() && ((SaveTemps && !InputIsBitcode) || EmbedBitcode))
return nullptr;
Inputs = &CJ->getInputs();
Modified: cfe/trunk/lib/Driver/ToolChains/Clang.cpp
URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Driver/ToolChains/Clang.cpp?rev=330194&r1=330193&r2=330194&view=diff
==============================================================================
--- cfe/trunk/lib/Driver/ToolChains/Clang.cpp (original)
+++ cfe/trunk/lib/Driver/ToolChains/Clang.cpp Tue Apr 17 09:39:25 2018
@@ -3267,6 +3267,9 @@ void Clang::ConstructJob(Compilation &C,
Args.AddLastArg(CmdArgs, options::OPT_fthinlto_index_EQ);
}
+ if (const Arg *A = Args.getLastArg(options::OPT_save_temps_EQ))
+ Args.AddLastArg(CmdArgs, options::OPT_save_temps_EQ);
+
// Embed-bitcode option.
if (C.getDriver().embedBitcodeInObject() && !C.getDriver().isUsingLTO() &&
(isa<BackendJobAction>(JA) || isa<AssembleJobAction>(JA))) {
Modified: cfe/trunk/lib/Frontend/CompilerInvocation.cpp
URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Frontend/CompilerInvocation.cpp?rev=330194&r1=330193&r2=330194&view=diff
==============================================================================
--- cfe/trunk/lib/Frontend/CompilerInvocation.cpp (original)
+++ cfe/trunk/lib/Frontend/CompilerInvocation.cpp Tue Apr 17 09:39:25 2018
@@ -507,7 +507,8 @@ static void setPGOUseInstrumentor(CodeGe
static bool ParseCodeGenArgs(CodeGenOptions &Opts, ArgList &Args, InputKind IK,
DiagnosticsEngine &Diags,
- const TargetOptions &TargetOpts) {
+ const TargetOptions &TargetOpts,
+ const FrontendOptions &FrontendOpts) {
bool Success = true;
llvm::Triple Triple = llvm::Triple(TargetOpts.Triple);
@@ -758,6 +759,12 @@ static bool ParseCodeGenArgs(CodeGenOpti
<< A->getAsString(Args) << "-x ir";
Opts.ThinLTOIndexFile = Args.getLastArgValue(OPT_fthinlto_index_EQ);
}
+ if (Arg *A = Args.getLastArg(OPT_save_temps_EQ))
+ Opts.SaveTempsFilePrefix =
+ llvm::StringSwitch<std::string>(A->getValue())
+ .Case("obj", FrontendOpts.OutputFile)
+ .Default(llvm::sys::path::filename(FrontendOpts.OutputFile).str());
+
Opts.ThinLinkBitcodeFile = Args.getLastArgValue(OPT_fthin_link_bitcode_EQ);
Opts.MSVolatile = Args.hasArg(OPT_fms_volatile);
@@ -2927,7 +2934,7 @@ bool CompilerInvocation::CreateFromArgs(
LangOpts.IsHeaderFile);
ParseTargetArgs(Res.getTargetOpts(), Args, Diags);
Success &= ParseCodeGenArgs(Res.getCodeGenOpts(), Args, DashX, Diags,
- Res.getTargetOpts());
+ Res.getTargetOpts(), Res.getFrontendOpts());
ParseHeaderSearchArgs(Res.getHeaderSearchOpts(), Args,
Res.getFileSystemOpts().WorkingDir);
if (DashX.getFormat() == InputKind::Precompiled ||
Modified: cfe/trunk/test/CodeGen/thinlto_backend.ll
URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/CodeGen/thinlto_backend.ll?rev=330194&r1=330193&r2=330194&view=diff
==============================================================================
--- cfe/trunk/test/CodeGen/thinlto_backend.ll (original)
+++ cfe/trunk/test/CodeGen/thinlto_backend.ll Tue Apr 17 09:39:25 2018
@@ -26,8 +26,16 @@
; RUN: %clang -target x86_64-unknown-linux-gnu -O2 -o %t4.o -x ir %t5.o -c -fthinlto-index=%t.thinlto.bc
; RUN: llvm-nm %t4.o | count 0
-; Ensure f2 was imported
-; RUN: %clang -target x86_64-unknown-linux-gnu -O2 -o %t3.o -x ir %t1.o -c -fthinlto-index=%t.thinlto.bc
+; Ensure f2 was imported. Check for all 3 flavors of -save-temps[=cwd|obj].
+; RUN: %clang -target x86_64-unknown-linux-gnu -O2 -o %t3.o -x ir %t1.o -c -fthinlto-index=%t.thinlto.bc -save-temps=obj
+; RUN: llvm-dis %t1.s.0.3.import.bc -o - | FileCheck --check-prefix=CHECK-IMPORT %s
+; RUN: mkdir -p %T/dir1
+; RUN: (cd %T/dir1 && %clang -target x86_64-unknown-linux-gnu -O2 -o %t3.o -x ir %t1.o -c -fthinlto-index=%t.thinlto.bc -save-temps=cwd)
+; RUN: llvm-dis %T/dir1/*1.s.0.3.import.bc -o - | FileCheck --check-prefix=CHECK-IMPORT %s
+; RUN: mkdir -p %T/dir2
+; RUN: (cd %T/dir2 && %clang -target x86_64-unknown-linux-gnu -O2 -o %t3.o -x ir %t1.o -c -fthinlto-index=%t.thinlto.bc -save-temps)
+; RUN: llvm-dis %T/dir2/*1.s.0.3.import.bc -o - | FileCheck --check-prefix=CHECK-IMPORT %s
+; CHECK-IMPORT: define available_externally void @f2()
; RUN: llvm-nm %t3.o | FileCheck --check-prefix=CHECK-OBJ %s
; CHECK-OBJ: T f1
; CHECK-OBJ-NOT: U f2
Modified: cfe/trunk/test/Driver/thinlto_backend.c
URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Driver/thinlto_backend.c?rev=330194&r1=330193&r2=330194&view=diff
==============================================================================
--- cfe/trunk/test/Driver/thinlto_backend.c (original)
+++ cfe/trunk/test/Driver/thinlto_backend.c Tue Apr 17 09:39:25 2018
@@ -5,6 +5,15 @@
// RUN: %clang -O2 -o %t1.o -x ir %t.o -c -fthinlto-index=%t.thinlto.bc -### 2>&1 | FileCheck %s -check-prefix=CHECK-THINLTOBE-ACTION
// CHECK-THINLTOBE-ACTION: -fthinlto-index=
+// -save-temps should be passed to cc1
+// RUN: %clang -O2 -o %t1.o -x ir %t.o -c -fthinlto-index=%t.thinlto.bc -save-temps -### 2>&1 | FileCheck %s -check-prefix=CHECK-SAVE-TEMPS -check-prefix=CHECK-SAVE-TEMPS-CWD
+// RUN: %clang -O2 -o %t1.o -x ir %t.o -c -fthinlto-index=%t.thinlto.bc -save-temps=cwd -### 2>&1 | FileCheck %s -check-prefix=CHECK-SAVE-TEMPS -check-prefix=CHECK-SAVE-TEMPS-CWD
+// RUN: %clang -O2 -o %t1.o -x ir %t.o -c -fthinlto-index=%t.thinlto.bc -save-temps=obj -### 2>&1 | FileCheck %s -check-prefix=CHECK-SAVE-TEMPS -check-prefix=CHECK-SAVE-TEMPS-OBJ
+// CHECK-SAVE-TEMPS-NOT: -emit-llvm-bc
+// CHECK-SAVE-TEMPS-CWD: -save-temps=cwd
+// CHECK-SAVE-TEMPS-OBJ: -save-temps=obj
+// CHECK-SAVE-TEMPS-NOT: -emit-llvm-bc
+
// Ensure clang driver gives the expected error for incorrect input type
// RUN: not %clang -O2 -o %t1.o %s -c -fthinlto-index=%t.thinlto.bc 2>&1 | FileCheck %s -check-prefix=CHECK-WARNING
// CHECK-WARNING: error: invalid argument '-fthinlto-index={{.*}}' only allowed with '-x ir'
More information about the cfe-commits
mailing list