<div dir="ltr">This change was causing MSan and UBSan <a href="http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/29937">failures</a> and reverted in r354953.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 26, 2019 at 2:36 PM Rong Xu via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Author: xur<br>
Date: Tue Feb 26 14:37:46 2019<br>
New Revision: 354930<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=354930&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project?rev=354930&view=rev</a><br>
Log:<br>
[PGO] Context sensitive PGO (part 1)<br>
<br>
Current PGO profile counts are not context sensitive. The branch probabilities<br>
for the inlined functions are kept the same for all call-sites, and they might<br>
be very different from the actual branch probabilities. These suboptimal<br>
profiles can greatly affect some downstream optimizations, in particular for<br>
the machine basic block placement optimization.<br>
<br>
In this patch, we propose to have a post-inline PGO instrumentation/use pass,<br>
which we called Context Sensitive PGO (CSPGO). For the users who want the best<br>
possible performance, they can perform a second round of PGO instrument/use on<br>
the top of the regular PGO. They will have two sets of profile counts. The<br>
first pass profile will be manly for inline, indirect-call promotion, and<br>
CGSCC simplification pass optimizations. The second pass profile is for<br>
post-inline optimizations and code-gen optimizations.<br>
<br>
A typical usage:<br>
// Regular PGO instrumentation and generate pass1 profile.<br>
> clang -O2 -fprofile-generate source.c -o gen<br>
> ./gen<br>
> llvm-profdata merge default.*profraw -o pass1.profdata<br>
// CSPGO instrumentation.<br>
> clang -O2 -fprofile-use=pass1.profdata -fcs-profile-generate -o gen2<br>
> ./gen2<br>
// Merge two sets of profiles<br>
> llvm-profdata merge default.*profraw pass1.profdata -o profile.profdata<br>
// Use the combined profile. Pass manager will invoke two PGO use passes.<br>
> clang -O2 -fprofile-use=profile.profdata -o use<br>
<br>
This change touches many components in the compiler. The reviewed patch<br>
(D54175) will committed in phrases.<br>
<br>
Differential Revision: <a href="https://reviews.llvm.org/D54175" rel="noreferrer" target="_blank">https://reviews.llvm.org/D54175</a><br>
<br>
Modified:<br>
llvm/trunk/include/llvm/InitializePasses.h<br>
llvm/trunk/include/llvm/LTO/Config.h<br>
llvm/trunk/include/llvm/LinkAllPasses.h<br>
llvm/trunk/include/llvm/ProfileData/InstrProf.h<br>
llvm/trunk/include/llvm/ProfileData/InstrProfData.inc<br>
llvm/trunk/include/llvm/Transforms/Instrumentation.h<br>
llvm/trunk/include/llvm/Transforms/Instrumentation/InstrProfiling.h<br>
llvm/trunk/include/llvm/Transforms/Instrumentation/PGOInstrumentation.h<br>
llvm/trunk/lib/Passes/PassBuilder.cpp<br>
llvm/trunk/lib/ProfileData/InstrProf.cpp<br>
llvm/trunk/lib/Transforms/Instrumentation/InstrProfiling.cpp<br>
llvm/trunk/lib/Transforms/Instrumentation/PGOInstrumentation.cpp<br>
llvm/trunk/tools/gold/gold-plugin.cpp<br>
llvm/trunk/tools/llvm-lto2/llvm-lto2.cpp<br>
<br>
Modified: llvm/trunk/include/llvm/InitializePasses.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/InitializePasses.h?rev=354930&r1=354929&r2=354930&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/InitializePasses.h?rev=354930&r1=354929&r2=354930&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/include/llvm/InitializePasses.h (original)<br>
+++ llvm/trunk/include/llvm/InitializePasses.h Tue Feb 26 14:37:46 2019<br>
@@ -299,6 +299,7 @@ void initializePEIPass(PassRegistry&);<br>
void initializePGOIndirectCallPromotionLegacyPassPass(PassRegistry&);<br>
void initializePGOInstrumentationGenLegacyPassPass(PassRegistry&);<br>
void initializePGOInstrumentationUseLegacyPassPass(PassRegistry&);<br>
+void initializePGOInstrumentationGenCreateVarLegacyPassPass(PassRegistry&);<br>
void initializePGOMemOPSizeOptLegacyPassPass(PassRegistry&);<br>
void initializePHIEliminationPass(PassRegistry&);<br>
void initializePartialInlinerLegacyPassPass(PassRegistry&);<br>
<br>
Modified: llvm/trunk/include/llvm/LTO/Config.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/LTO/Config.h?rev=354930&r1=354929&r2=354930&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/LTO/Config.h?rev=354930&r1=354929&r2=354930&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/include/llvm/LTO/Config.h (original)<br>
+++ llvm/trunk/include/llvm/LTO/Config.h Tue Feb 26 14:37:46 2019<br>
@@ -55,6 +55,9 @@ struct Config {<br>
/// Disable entirely the optimizer, including importing for ThinLTO<br>
bool CodeGenOnly = false;<br>
<br>
+ /// Run PGO context sensitive IR instrumentation.<br>
+ bool RunCSIRInstr = false;<br>
+<br>
/// If this field is set, the set of passes run in the middle-end optimizer<br>
/// will be the one specified by the string. Only works with the new pass<br>
/// manager as the old one doesn't have this ability.<br>
@@ -73,6 +76,9 @@ struct Config {<br>
/// with this triple.<br>
std::string DefaultTriple;<br>
<br>
+ /// Context Sensitive PGO profile path.<br>
+ std::string CSIRProfile;<br>
+<br>
/// Sample PGO profile path.<br>
std::string SampleProfile;<br>
<br>
<br>
Modified: llvm/trunk/include/llvm/LinkAllPasses.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/LinkAllPasses.h?rev=354930&r1=354929&r2=354930&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/LinkAllPasses.h?rev=354930&r1=354929&r2=354930&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/include/llvm/LinkAllPasses.h (original)<br>
+++ llvm/trunk/include/llvm/LinkAllPasses.h Tue Feb 26 14:37:46 2019<br>
@@ -102,6 +102,7 @@ namespace {<br>
(void) llvm::createGCOVProfilerPass();<br>
(void) llvm::createPGOInstrumentationGenLegacyPass();<br>
(void) llvm::createPGOInstrumentationUseLegacyPass();<br>
+ (void) llvm::createPGOInstrumentationGenCreateVarLegacyPass();<br>
(void) llvm::createPGOIndirectCallPromotionLegacyPass();<br>
(void) llvm::createPGOMemOPSizeOptLegacyPass();<br>
(void) llvm::createInstrProfilingLegacyPass();<br>
<br>
Modified: llvm/trunk/include/llvm/ProfileData/InstrProf.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/ProfileData/InstrProf.h?rev=354930&r1=354929&r2=354930&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/ProfileData/InstrProf.h?rev=354930&r1=354929&r2=354930&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/include/llvm/ProfileData/InstrProf.h (original)<br>
+++ llvm/trunk/include/llvm/ProfileData/InstrProf.h Tue Feb 26 14:37:46 2019<br>
@@ -767,10 +767,20 @@ struct NamedInstrProfRecord : InstrProfR<br>
StringRef Name;<br>
uint64_t Hash;<br>
<br>
+ // We reserve this bit as the flag for context sensitive profile record.<br>
+ static const int CS_FLAG_IN_FUNC_HASH = 60;<br>
+<br>
NamedInstrProfRecord() = default;<br>
NamedInstrProfRecord(StringRef Name, uint64_t Hash,<br>
std::vector<uint64_t> Counts)<br>
: InstrProfRecord(std::move(Counts)), Name(Name), Hash(Hash) {}<br>
+<br>
+ static bool hasCSFlagInHash(uint64_t FuncHash) {<br>
+ return ((FuncHash >> CS_FLAG_IN_FUNC_HASH) & 1);<br>
+ }<br>
+ static void setCSFlagInHash(uint64_t &FuncHash) {<br>
+ FuncHash |= ((uint64_t)1 << CS_FLAG_IN_FUNC_HASH);<br>
+ }<br>
};<br>
<br>
uint32_t InstrProfRecord::getNumValueKinds() const {<br>
@@ -1004,6 +1014,8 @@ namespace RawInstrProf {<br>
// from control data struct is changed from raw pointer to Name's MD5 value.<br>
// Version 4: ValueDataBegin and ValueDataSizes fields are removed from the<br>
// raw header.<br>
+// Version 5: Bit 60 of FuncHash is reserved for the flag for the context<br>
+// sensitive records.<br>
const uint64_t Version = INSTR_PROF_RAW_VERSION;<br>
<br>
template <class IntPtrT> inline uint64_t getMagic();<br>
@@ -1040,6 +1052,10 @@ struct Header {<br>
void getMemOPSizeRangeFromOption(StringRef Str, int64_t &RangeStart,<br>
int64_t &RangeLast);<br>
<br>
+// Create a COMDAT variable INSTR_PROF_RAW_VERSION_VAR to make the runtime<br>
+// aware this is an ir_level profile so it can set the version flag.<br>
+void createIRLevelProfileFlagVar(Module &M, bool IsCS);<br>
+<br>
// Create the variable for the profile file name.<br>
void createProfileFileNameVar(Module &M, StringRef InstrProfileOutput);<br>
<br>
<br>
Modified: llvm/trunk/include/llvm/ProfileData/InstrProfData.inc<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/ProfileData/InstrProfData.inc?rev=354930&r1=354929&r2=354930&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/ProfileData/InstrProfData.inc?rev=354930&r1=354929&r2=354930&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/include/llvm/ProfileData/InstrProfData.inc (original)<br>
+++ llvm/trunk/include/llvm/ProfileData/InstrProfData.inc Tue Feb 26 14:37:46 2019<br>
@@ -635,10 +635,12 @@ serializeValueProfDataFrom(ValueProfReco<br>
* version for other variants of profile. We set the lowest bit of the upper 8<br>
* bits (i.e. bit 56) to 1 to indicate if this is an IR-level instrumentaiton<br>
* generated profile, and 0 if this is a Clang FE generated profile.<br>
+ * 1 in bit 57 indicates there are context-sensitive records in the profile.<br>
*/<br>
#define VARIANT_MASKS_ALL 0xff00000000000000ULL<br>
#define GET_VERSION(V) ((V) & ~VARIANT_MASKS_ALL)<br>
#define VARIANT_MASK_IR_PROF (0x1ULL << 56)<br>
+#define VARIANT_MASK_CSIR_PROF (0x1ULL << 57)<br>
#define INSTR_PROF_RAW_VERSION_VAR __llvm_profile_raw_version<br>
#define INSTR_PROF_PROFILE_RUNTIME_VAR __llvm_profile_runtime<br>
<br>
<br>
Modified: llvm/trunk/include/llvm/Transforms/Instrumentation.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Transforms/Instrumentation.h?rev=354930&r1=354929&r2=354930&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Transforms/Instrumentation.h?rev=354930&r1=354929&r2=354930&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/include/llvm/Transforms/Instrumentation.h (original)<br>
+++ llvm/trunk/include/llvm/Transforms/Instrumentation.h Tue Feb 26 14:37:46 2019<br>
@@ -87,10 +87,14 @@ struct GCOVOptions {<br>
ModulePass *createGCOVProfilerPass(const GCOVOptions &Options =<br>
GCOVOptions::getDefault());<br>
<br>
-// PGO Instrumention<br>
-ModulePass *createPGOInstrumentationGenLegacyPass();<br>
+// PGO Instrumention. Parameter IsCS indicates if this is the context senstive<br>
+// instrumentation.<br>
+ModulePass *createPGOInstrumentationGenLegacyPass(bool IsCS = false);<br>
ModulePass *<br>
-createPGOInstrumentationUseLegacyPass(StringRef Filename = StringRef(""));<br>
+createPGOInstrumentationUseLegacyPass(StringRef Filename = StringRef(""),<br>
+ bool IsCS = false);<br>
+ModulePass *createPGOInstrumentationGenCreateVarLegacyPass(<br>
+ StringRef CSInstrName = StringRef(""));<br>
ModulePass *createPGOIndirectCallPromotionLegacyPass(bool InLTO = false,<br>
bool SamplePGO = false);<br>
FunctionPass *createPGOMemOPSizeOptLegacyPass();<br>
@@ -132,15 +136,19 @@ struct InstrProfOptions {<br>
// Use atomic profile counter increments.<br>
bool Atomic = false;<br>
<br>
+ // Use BFI to guide register promotion<br>
+ bool UseBFIInPromotion = false;<br>
+<br>
// Name of the profile file to use as output<br>
std::string InstrProfileOutput;<br>
<br>
InstrProfOptions() = default;<br>
};<br>
<br>
-/// Insert frontend instrumentation based profiling.<br>
+/// Insert frontend instrumentation based profiling. Parameter IsCS indicates if<br>
+// this is the context senstive instrumentation.<br>
ModulePass *createInstrProfilingLegacyPass(<br>
- const InstrProfOptions &Options = InstrProfOptions());<br>
+ const InstrProfOptions &Options = InstrProfOptions(), bool IsCS = false);<br>
<br>
FunctionPass *createHWAddressSanitizerPass(bool CompileKernel = false,<br>
bool Recover = false);<br>
<br>
Modified: llvm/trunk/include/llvm/Transforms/Instrumentation/InstrProfiling.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Transforms/Instrumentation/InstrProfiling.h?rev=354930&r1=354929&r2=354930&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Transforms/Instrumentation/InstrProfiling.h?rev=354930&r1=354929&r2=354930&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/include/llvm/Transforms/Instrumentation/InstrProfiling.h (original)<br>
+++ llvm/trunk/include/llvm/Transforms/Instrumentation/InstrProfiling.h Tue Feb 26 14:37:46 2019<br>
@@ -35,7 +35,8 @@ using LoadStorePair = std::pair<Instruct<br>
class InstrProfiling : public PassInfoMixin<InstrProfiling> {<br>
public:<br>
InstrProfiling() = default;<br>
- InstrProfiling(const InstrProfOptions &Options) : Options(Options) {}<br>
+ InstrProfiling(const InstrProfOptions &Options, bool IsCS)<br>
+ : Options(Options), IsCS(IsCS) {}<br>
<br>
PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM);<br>
bool run(Module &M, const TargetLibraryInfo &TLI);<br>
@@ -60,6 +61,9 @@ private:<br>
GlobalVariable *NamesVar;<br>
size_t NamesSize;<br>
<br>
+ // Is this lowering for the context-sensitive instrumentation.<br>
+ bool IsCS;<br>
+<br>
// vector of counter load/store pairs to be register promoted.<br>
std::vector<LoadStorePair> PromotionCandidates;<br>
<br>
<br>
Modified: llvm/trunk/include/llvm/Transforms/Instrumentation/PGOInstrumentation.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Transforms/Instrumentation/PGOInstrumentation.h?rev=354930&r1=354929&r2=354930&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Transforms/Instrumentation/PGOInstrumentation.h?rev=354930&r1=354929&r2=354930&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/include/llvm/Transforms/Instrumentation/PGOInstrumentation.h (original)<br>
+++ llvm/trunk/include/llvm/Transforms/Instrumentation/PGOInstrumentation.h Tue Feb 26 14:37:46 2019<br>
@@ -17,6 +17,7 @@<br>
<br>
#include "llvm/ADT/ArrayRef.h"<br>
#include "llvm/IR/PassManager.h"<br>
+#include "llvm/ProfileData/InstrProf.h"<br>
#include <cstdint><br>
#include <string><br>
<br>
@@ -27,22 +28,50 @@ class Instruction;<br>
class Module;<br>
<br>
/// The instrumentation (profile-instr-gen) pass for IR based PGO.<br>
+// We use this pass to create COMDAT profile variables for context<br>
+// sensitive PGO (CSPGO). The reason to have a pass for this is CSPGO<br>
+// can be run after LTO/ThinLTO linking. Lld linker needs to see<br>
+// all the COMDAT variables before linking. So we have this pass<br>
+// always run before linking for CSPGO.<br>
+class PGOInstrumentationGenCreateVar<br>
+ : public PassInfoMixin<PGOInstrumentationGenCreateVar> {<br>
+public:<br>
+ PGOInstrumentationGenCreateVar(std::string CSInstrName = "")<br>
+ : CSInstrName(CSInstrName) {}<br>
+ PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM) {<br>
+ createProfileFileNameVar(M, CSInstrName);<br>
+ createIRLevelProfileFlagVar(M, /* IsCS */ true);<br>
+ return PreservedAnalyses::all();<br>
+ }<br>
+<br>
+private:<br>
+ std::string CSInstrName;<br>
+};<br>
+<br>
+/// The instrumentation (profile-instr-gen) pass for IR based PGO.<br>
class PGOInstrumentationGen : public PassInfoMixin<PGOInstrumentationGen> {<br>
public:<br>
+ PGOInstrumentationGen(bool IsCS = false) : IsCS(IsCS) {}<br>
PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM);<br>
+<br>
+private:<br>
+ // If this is a context sensitive instrumentation.<br>
+ bool IsCS;<br>
};<br>
<br>
/// The profile annotation (profile-instr-use) pass for IR based PGO.<br>
class PGOInstrumentationUse : public PassInfoMixin<PGOInstrumentationUse> {<br>
public:<br>
PGOInstrumentationUse(std::string Filename = "",<br>
- std::string RemappingFilename = "");<br>
+ std::string RemappingFilename = "", bool IsCS = false);<br>
<br>
PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM);<br>
<br>
private:<br>
std::string ProfileFileName;<br>
std::string ProfileRemappingFileName;<br>
+ // If this is a context sensitive instrumentation.<br>
+ bool IsCS;<br>
};<br>
<br>
/// The indirect function call promotion pass.<br>
<br>
Modified: llvm/trunk/lib/Passes/PassBuilder.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Passes/PassBuilder.cpp?rev=354930&r1=354929&r2=354930&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Passes/PassBuilder.cpp?rev=354930&r1=354929&r2=354930&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Passes/PassBuilder.cpp (original)<br>
+++ llvm/trunk/lib/Passes/PassBuilder.cpp Tue Feb 26 14:37:46 2019<br>
@@ -569,7 +569,8 @@ void PassBuilder::addPGOInstrPasses(Modu<br>
if (!ProfileGenFile.empty())<br>
Options.InstrProfileOutput = ProfileGenFile;<br>
Options.DoCounterPromotion = true;<br>
- MPM.addPass(InstrProfiling(Options));<br>
+ Options.UseBFIInPromotion = false;<br>
+ MPM.addPass(InstrProfiling(Options, false));<br>
}<br>
<br>
if (!ProfileUseFile.empty())<br>
<br>
Modified: llvm/trunk/lib/ProfileData/InstrProf.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/ProfileData/InstrProf.cpp?rev=354930&r1=354929&r2=354930&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/ProfileData/InstrProf.cpp?rev=354930&r1=354929&r2=354930&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/ProfileData/InstrProf.cpp (original)<br>
+++ llvm/trunk/lib/ProfileData/InstrProf.cpp Tue Feb 26 14:37:46 2019<br>
@@ -1011,6 +1011,25 @@ void getMemOPSizeRangeFromOption(StringR<br>
assert(RangeLast >= RangeStart);<br>
}<br>
<br>
+// Create a COMDAT variable INSTR_PROF_RAW_VERSION_VAR to make the runtime<br>
+// aware this is an ir_level profile so it can set the version flag.<br>
+void createIRLevelProfileFlagVar(Module &M, bool IsCS) {<br>
+ const StringRef VarName(INSTR_PROF_QUOTE(INSTR_PROF_RAW_VERSION_VAR));<br>
+ Type *IntTy64 = Type::getInt64Ty(M.getContext());<br>
+ uint64_t ProfileVersion = (INSTR_PROF_RAW_VERSION | VARIANT_MASK_IR_PROF);<br>
+ if (IsCS)<br>
+ ProfileVersion |= VARIANT_MASK_CSIR_PROF;<br>
+ auto IRLevelVersionVariable = new GlobalVariable(<br>
+ M, IntTy64, true, GlobalValue::WeakAnyLinkage,<br>
+ Constant::getIntegerValue(IntTy64, APInt(64, ProfileVersion)), VarName);<br>
+ IRLevelVersionVariable->setVisibility(GlobalValue::DefaultVisibility);<br>
+ Triple TT(M.getTargetTriple());<br>
+ if (TT.supportsCOMDAT()) {<br>
+ IRLevelVersionVariable->setLinkage(GlobalValue::ExternalLinkage);<br>
+ IRLevelVersionVariable->setComdat(M.getOrInsertComdat(VarName));<br>
+ }<br>
+}<br>
+<br>
// Create the variable for the profile file name.<br>
void createProfileFileNameVar(Module &M, StringRef InstrProfileOutput) {<br>
if (InstrProfileOutput.empty())<br>
<br>
Modified: llvm/trunk/lib/Transforms/Instrumentation/InstrProfiling.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Instrumentation/InstrProfiling.cpp?rev=354930&r1=354929&r2=354930&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Instrumentation/InstrProfiling.cpp?rev=354930&r1=354929&r2=354930&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Transforms/Instrumentation/InstrProfiling.cpp (original)<br>
+++ llvm/trunk/lib/Transforms/Instrumentation/InstrProfiling.cpp Tue Feb 26 14:37:46 2019<br>
@@ -18,6 +18,8 @@<br>
#include "llvm/ADT/StringRef.h"<br>
#include "llvm/ADT/Triple.h"<br>
#include "llvm/ADT/Twine.h"<br>
+#include "llvm/Analysis/BlockFrequencyInfo.h"<br>
+#include "llvm/Analysis/BranchProbabilityInfo.h"<br>
#include "llvm/Analysis/LoopInfo.h"<br>
#include "llvm/Analysis/TargetLibraryInfo.h"<br>
#include "llvm/IR/Attributes.h"<br>
@@ -147,8 +149,8 @@ public:<br>
static char ID;<br>
<br>
InstrProfilingLegacyPass() : ModulePass(ID) {}<br>
- InstrProfilingLegacyPass(const InstrProfOptions &Options)<br>
- : ModulePass(ID), InstrProf(Options) {}<br>
+ InstrProfilingLegacyPass(const InstrProfOptions &Options, bool IsCS)<br>
+ : ModulePass(ID), InstrProf(Options, IsCS) {}<br>
<br>
StringRef getPassName() const override {<br>
return "Frontend instrumentation-based coverage lowering";<br>
@@ -232,9 +234,9 @@ class PGOCounterPromoter {<br>
public:<br>
PGOCounterPromoter(<br>
DenseMap<Loop *, SmallVector<LoadStorePair, 8>> &LoopToCands,<br>
- Loop &CurLoop, LoopInfo &LI)<br>
+ Loop &CurLoop, LoopInfo &LI, BlockFrequencyInfo *BFI)<br>
: LoopToCandidates(LoopToCands), ExitBlocks(), InsertPts(), L(CurLoop),<br>
- LI(LI) {<br>
+ LI(LI), BFI(BFI) {<br>
<br>
SmallVector<BasicBlock *, 8> LoopExitBlocks;<br>
SmallPtrSet<BasicBlock *, 8> BlockSet;<br>
@@ -263,6 +265,20 @@ public:<br>
SSAUpdater SSA(&NewPHIs);<br>
Value *InitVal = ConstantInt::get(Cand.first->getType(), 0);<br>
<br>
+ // If BFI is set, we will use it to guide the promotions.<br>
+ if (BFI) {<br>
+ auto *BB = Cand.first->getParent();<br>
+ auto InstrCount = BFI->getBlockProfileCount(BB);<br>
+ if (!InstrCount)<br>
+ continue;<br>
+ auto PreheaderCount = BFI->getBlockProfileCount(L.getLoopPreheader());<br>
+ // If the average loop trip count is not greater than 1.5, we skip<br>
+ // promotion.<br>
+ if (PreheaderCount &&<br>
+ (PreheaderCount.getValue() * 3) >= (InstrCount.getValue() * 2))<br>
+ continue;<br>
+ }<br>
+<br>
PGOCounterPromoterHelper Promoter(Cand.first, Cand.second, SSA, InitVal,<br>
L.getLoopPreheader(), ExitBlocks,<br>
InsertPts, LoopToCandidates, LI);<br>
@@ -312,6 +328,11 @@ private:<br>
<br>
SmallVector<BasicBlock *, 8> ExitingBlocks;<br>
LP->getExitingBlocks(ExitingBlocks);<br>
+<br>
+ // If BFI is set, we do more aggressive promotions based on BFI.<br>
+ if (BFI)<br>
+ return (unsigned)-1;<br>
+<br>
// Not considierered speculative.<br>
if (ExitingBlocks.size() == 1)<br>
return MaxNumOfPromotionsPerLoop;<br>
@@ -343,6 +364,7 @@ private:<br>
SmallVector<Instruction *, 8> InsertPts;<br>
Loop &L;<br>
LoopInfo &LI;<br>
+ BlockFrequencyInfo *BFI;<br>
};<br>
<br>
} // end anonymous namespace<br>
@@ -365,8 +387,9 @@ INITIALIZE_PASS_END(<br>
"Frontend instrumentation-based coverage lowering.", false, false)<br>
<br>
ModulePass *<br>
-llvm::createInstrProfilingLegacyPass(const InstrProfOptions &Options) {<br>
- return new InstrProfilingLegacyPass(Options);<br>
+llvm::createInstrProfilingLegacyPass(const InstrProfOptions &Options,<br>
+ bool IsCS) {<br>
+ return new InstrProfilingLegacyPass(Options, IsCS);<br>
}<br>
<br>
static InstrProfIncrementInst *castToIncrementInst(Instruction *Instr) {<br>
@@ -415,6 +438,13 @@ void InstrProfiling::promoteCounterLoadS<br>
LoopInfo LI(DT);<br>
DenseMap<Loop *, SmallVector<LoadStorePair, 8>> LoopPromotionCandidates;<br>
<br>
+ std::unique_ptr<BlockFrequencyInfo> BFI;<br>
+ if (Options.UseBFIInPromotion) {<br>
+ std::unique_ptr<BranchProbabilityInfo> BPI;<br>
+ BPI.reset(new BranchProbabilityInfo(*F, LI, TLI));<br>
+ BFI.reset(new BlockFrequencyInfo(*F, *BPI, LI));<br>
+ }<br>
+<br>
for (const auto &LoadStore : PromotionCandidates) {<br>
auto *CounterLoad = LoadStore.first;<br>
auto *CounterStore = LoadStore.second;<br>
@@ -430,7 +460,7 @@ void InstrProfiling::promoteCounterLoadS<br>
// Do a post-order traversal of the loops so that counter updates can be<br>
// iteratively hoisted outside the loop nest.<br>
for (auto *Loop : llvm::reverse(Loops)) {<br>
- PGOCounterPromoter Promoter(LoopPromotionCandidates, *Loop, LI);<br>
+ PGOCounterPromoter Promoter(LoopPromotionCandidates, *Loop, LI, BFI.get());<br>
Promoter.run(&TotalCountersPromoted);<br>
}<br>
}<br>
@@ -681,7 +711,6 @@ static bool needsRuntimeRegistrationOfSe<br>
// Don't do this for Darwin. compiler-rt uses linker magic.<br>
if (TT.isOSDarwin())<br>
return false;<br>
-<br>
// Use linker script magic to get data/cnts/name start/end.<br>
if (TT.isOSLinux() || TT.isOSFreeBSD() || TT.isOSNetBSD() ||<br>
TT.isOSFuchsia() || TT.isPS4CPU() || TT.isOSWindows())<br>
@@ -985,8 +1014,12 @@ void InstrProfiling::emitUses() {<br>
}<br>
<br>
void InstrProfiling::emitInitialization() {<br>
- // Create variable for profile name.<br>
- createProfileFileNameVar(*M, Options.InstrProfileOutput);<br>
+ // Create ProfileFileName variable. Don't don't this for the<br>
+ // context-sensitive instrumentation lowering: This lowering is after<br>
+ // LTO/ThinLTO linking. Pass PGOInstrumentationGenCreateVar should<br>
+ // have already create the variable before LTO/ThinLTO linking.<br>
+ if (!IsCS)<br>
+ createProfileFileNameVar(*M, Options.InstrProfileOutput);<br>
Function *RegisterF = M->getFunction(getInstrProfRegFuncsName());<br>
if (!RegisterF)<br>
return;<br>
<br>
Modified: llvm/trunk/lib/Transforms/Instrumentation/PGOInstrumentation.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Instrumentation/PGOInstrumentation.cpp?rev=354930&r1=354929&r2=354930&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Instrumentation/PGOInstrumentation.cpp?rev=354930&r1=354929&r2=354930&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Transforms/Instrumentation/PGOInstrumentation.cpp (original)<br>
+++ llvm/trunk/lib/Transforms/Instrumentation/PGOInstrumentation.cpp Tue Feb 26 14:37:46 2019<br>
@@ -65,6 +65,7 @@<br>
#include "llvm/Analysis/IndirectCallVisitor.h"<br>
#include "llvm/Analysis/LoopInfo.h"<br>
#include "llvm/Analysis/OptimizationRemarkEmitter.h"<br>
+#include "llvm/Analysis/ProfileSummaryInfo.h"<br>
#include "llvm/IR/Attributes.h"<br>
#include "llvm/IR/BasicBlock.h"<br>
#include "llvm/IR/CFG.h"<br>
@@ -132,6 +133,19 @@ STATISTIC(NumOfPGOFunc, "Number of funct<br>
STATISTIC(NumOfPGOMismatch, "Number of functions having mismatch profile.");<br>
STATISTIC(NumOfPGOMissing, "Number of functions without profile.");<br>
STATISTIC(NumOfPGOICall, "Number of indirect call value instrumentations.");<br>
+STATISTIC(NumOfCSPGOInstrument, "Number of edges instrumented in CSPGO.");<br>
+STATISTIC(NumOfCSPGOSelectInsts,<br>
+ "Number of select instruction instrumented in CSPGO.");<br>
+STATISTIC(NumOfCSPGOMemIntrinsics,<br>
+ "Number of mem intrinsics instrumented in CSPGO.");<br>
+STATISTIC(NumOfCSPGOEdge, "Number of edges in CSPGO.");<br>
+STATISTIC(NumOfCSPGOBB, "Number of basic-blocks in CSPGO.");<br>
+STATISTIC(NumOfCSPGOSplit, "Number of critical edge splits in CSPGO.");<br>
+STATISTIC(NumOfCSPGOFunc,<br>
+ "Number of functions having valid profile counts in CSPGO.");<br>
+STATISTIC(NumOfCSPGOMismatch,<br>
+ "Number of functions having mismatch profile in CSPGO.");<br>
+STATISTIC(NumOfCSPGOMissing, "Number of functions without profile in CSPGO.");<br>
<br>
// Command line option to specify the file to read profile from. This is<br>
// mainly used for testing.<br>
@@ -383,7 +397,8 @@ class PGOInstrumentationGenLegacyPass :<br>
public:<br>
static char ID;<br>
<br>
- PGOInstrumentationGenLegacyPass() : ModulePass(ID) {<br>
+ PGOInstrumentationGenLegacyPass(bool IsCS = false)<br>
+ : ModulePass(ID), IsCS(IsCS) {<br>
initializePGOInstrumentationGenLegacyPassPass(<br>
*PassRegistry::getPassRegistry());<br>
}<br>
@@ -391,6 +406,8 @@ public:<br>
StringRef getPassName() const override { return "PGOInstrumentationGenPass"; }<br>
<br>
private:<br>
+ // Is this is context-sensitive instrumentation.<br>
+ bool IsCS;<br>
bool runOnModule(Module &M) override;<br>
<br>
void getAnalysisUsage(AnalysisUsage &AU) const override {<br>
@@ -403,8 +420,8 @@ public:<br>
static char ID;<br>
<br>
// Provide the profile filename as the parameter.<br>
- PGOInstrumentationUseLegacyPass(std::string Filename = "")<br>
- : ModulePass(ID), ProfileFileName(std::move(Filename)) {<br>
+ PGOInstrumentationUseLegacyPass(std::string Filename = "", bool IsCS = false)<br>
+ : ModulePass(ID), ProfileFileName(std::move(Filename)), IsCS(IsCS) {<br>
if (!PGOTestProfileFile.empty())<br>
ProfileFileName = PGOTestProfileFile;<br>
initializePGOInstrumentationUseLegacyPassPass(<br>
@@ -415,14 +432,38 @@ public:<br>
<br>
private:<br>
std::string ProfileFileName;<br>
+ // Is this is context-sensitive instrumentation use.<br>
+ bool IsCS;<br>
<br>
bool runOnModule(Module &M) override;<br>
<br>
void getAnalysisUsage(AnalysisUsage &AU) const override {<br>
+ AU.addRequired<ProfileSummaryInfoWrapperPass>();<br>
AU.addRequired<BlockFrequencyInfoWrapperPass>();<br>
}<br>
};<br>
<br>
+class PGOInstrumentationGenCreateVarLegacyPass : public ModulePass {<br>
+public:<br>
+ static char ID;<br>
+ StringRef getPassName() const override {<br>
+ return "PGOInstrumentationGenCreateVarPass";<br>
+ }<br>
+ PGOInstrumentationGenCreateVarLegacyPass(std::string CSInstrName = "")<br>
+ : ModulePass(ID), InstrProfileOutput(CSInstrName) {<br>
+ initializePGOInstrumentationGenCreateVarLegacyPassPass(<br>
+ *PassRegistry::getPassRegistry());<br>
+ }<br>
+<br>
+private:<br>
+ bool runOnModule(Module &M) override {<br>
+ createProfileFileNameVar(M, InstrProfileOutput);<br>
+ createIRLevelProfileFlagVar(M, true);<br>
+ return false;<br>
+ }<br>
+ std::string InstrProfileOutput;<br>
+};<br>
+<br>
} // end anonymous namespace<br>
<br>
char PGOInstrumentationGenLegacyPass::ID = 0;<br>
@@ -434,8 +475,8 @@ INITIALIZE_PASS_DEPENDENCY(BranchProbabi<br>
INITIALIZE_PASS_END(PGOInstrumentationGenLegacyPass, "pgo-instr-gen",<br>
"PGO instrumentation.", false, false)<br>
<br>
-ModulePass *llvm::createPGOInstrumentationGenLegacyPass() {<br>
- return new PGOInstrumentationGenLegacyPass();<br>
+ModulePass *llvm::createPGOInstrumentationGenLegacyPass(bool IsCS) {<br>
+ return new PGOInstrumentationGenLegacyPass(IsCS);<br>
}<br>
<br>
char PGOInstrumentationUseLegacyPass::ID = 0;<br>
@@ -444,11 +485,25 @@ INITIALIZE_PASS_BEGIN(PGOInstrumentation<br>
"Read PGO instrumentation profile.", false, false)<br>
INITIALIZE_PASS_DEPENDENCY(BlockFrequencyInfoWrapperPass)<br>
INITIALIZE_PASS_DEPENDENCY(BranchProbabilityInfoWrapperPass)<br>
+INITIALIZE_PASS_DEPENDENCY(ProfileSummaryInfoWrapperPass)<br>
INITIALIZE_PASS_END(PGOInstrumentationUseLegacyPass, "pgo-instr-use",<br>
"Read PGO instrumentation profile.", false, false)<br>
<br>
-ModulePass *llvm::createPGOInstrumentationUseLegacyPass(StringRef Filename) {<br>
- return new PGOInstrumentationUseLegacyPass(Filename.str());<br>
+ModulePass *llvm::createPGOInstrumentationUseLegacyPass(StringRef Filename,<br>
+ bool IsCS) {<br>
+ return new PGOInstrumentationUseLegacyPass(Filename.str(), IsCS);<br>
+}<br>
+<br>
+char PGOInstrumentationGenCreateVarLegacyPass::ID = 0;<br>
+<br>
+INITIALIZE_PASS(PGOInstrumentationGenCreateVarLegacyPass,<br>
+ "pgo-instr-gen-create-var",<br>
+ "Create PGO instrumentation version variable for CSPGO.", false,<br>
+ false)<br>
+<br>
+ModulePass *<br>
+llvm::createPGOInstrumentationGenCreateVarLegacyPass(StringRef CSInstrName) {<br>
+ return new PGOInstrumentationGenCreateVarLegacyPass(CSInstrName);<br>
}<br>
<br>
namespace {<br>
@@ -496,6 +551,9 @@ template <class Edge, class BBInfo> clas<br>
private:<br>
Function &F;<br>
<br>
+ // Is this is context-sensitive instrumentation.<br>
+ bool IsCS;<br>
+<br>
// A map that stores the Comdat group in function F.<br>
std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers;<br>
<br>
@@ -535,15 +593,23 @@ public:<br>
Function &Func,<br>
std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers,<br>
bool CreateGlobalVar = false, BranchProbabilityInfo *BPI = nullptr,<br>
- BlockFrequencyInfo *BFI = nullptr)<br>
- : F(Func), ComdatMembers(ComdatMembers), ValueSites(IPVK_Last + 1),<br>
- SIVisitor(Func), MIVisitor(Func), MST(F, BPI, BFI) {<br>
+ BlockFrequencyInfo *BFI = nullptr, bool IsCS = false)<br>
+ : F(Func), IsCS(IsCS), ComdatMembers(ComdatMembers),<br>
+ ValueSites(IPVK_Last + 1), SIVisitor(Func), MIVisitor(Func),<br>
+ MST(F, BPI, BFI) {<br>
// This should be done before CFG hash computation.<br>
SIVisitor.countSelects(Func);<br>
MIVisitor.countMemIntrinsics(Func);<br>
- NumOfPGOSelectInsts += SIVisitor.getNumOfSelectInsts();<br>
- NumOfPGOMemIntrinsics += MIVisitor.getNumOfMemIntrinsics();<br>
- ValueSites[IPVK_IndirectCallTarget] = findIndirectCalls(Func);<br>
+ if (!IsCS) {<br>
+ NumOfPGOSelectInsts += SIVisitor.getNumOfSelectInsts();<br>
+ NumOfPGOMemIntrinsics += MIVisitor.getNumOfMemIntrinsics();<br>
+ NumOfPGOBB += MST.BBInfos.size();<br>
+ ValueSites[IPVK_IndirectCallTarget] = findIndirectCalls(Func);<br>
+ } else {<br>
+ NumOfCSPGOSelectInsts += SIVisitor.getNumOfSelectInsts();<br>
+ NumOfCSPGOMemIntrinsics += MIVisitor.getNumOfMemIntrinsics();<br>
+ NumOfCSPGOBB += MST.BBInfos.size();<br>
+ }<br>
ValueSites[IPVK_MemOPSize] = MIVisitor.findMemIntrinsics(Func);<br>
<br>
FuncName = getPGOFuncName(F);<br>
@@ -552,13 +618,12 @@ public:<br>
renameComdatFunction();<br>
LLVM_DEBUG(dumpInfo("after CFGMST"));<br>
<br>
- NumOfPGOBB += MST.BBInfos.size();<br>
for (auto &E : MST.AllEdges) {<br>
if (E->Removed)<br>
continue;<br>
- NumOfPGOEdge++;<br>
+ IsCS ? NumOfCSPGOEdge++ : NumOfPGOEdge++;<br>
if (!E->InMST)<br>
- NumOfPGOInstrument++;<br>
+ IsCS ? NumOfCSPGOInstrument++ : NumOfPGOInstrument++;<br>
}<br>
<br>
if (CreateGlobalVar)<br>
@@ -597,9 +662,17 @@ void FuncPGOInstrumentation<Edge, BBInfo<br>
}<br>
}<br>
JC.update(Indexes);<br>
+<br>
+ // Hash format for context sensitive profile. Reserve 4 bits for other<br>
+ // information.<br>
FunctionHash = (uint64_t)SIVisitor.getNumOfSelectInsts() << 56 |<br>
(uint64_t)ValueSites[IPVK_IndirectCallTarget].size() << 48 |<br>
+ //(uint64_t)ValueSites[IPVK_MemOPSize].size() << 40 |<br>
(uint64_t)MST.AllEdges.size() << 32 | JC.getCRC();<br>
+ // Reserve bit 60-63 for other information purpose.<br>
+ FunctionHash &= 0x0FFFFFFFFFFFFFFF;<br>
+ if (IsCS)<br>
+ NamedInstrProfRecord::setCSFlagInHash(FunctionHash);<br>
LLVM_DEBUG(dbgs() << "Function Hash Computation for " << F.getName() << ":\n"<br>
<< " CRC = " << JC.getCRC()<br>
<< ", Selects = " << SIVisitor.getNumOfSelectInsts()<br>
@@ -705,7 +778,7 @@ BasicBlock *FuncPGOInstrumentation<Edge,<br>
<br>
// For a critical edge, we have to split. Instrument the newly<br>
// created BB.<br>
- NumOfPGOSplit++;<br>
+ IsCS ? NumOfCSPGOSplit++ : NumOfPGOSplit++;<br>
LLVM_DEBUG(dbgs() << "Split critical edge: " << getBBInfo(SrcBB).Index<br>
<< " --> " << getBBInfo(DestBB).Index << "\n");<br>
unsigned SuccNum = GetSuccessorNumber(SrcBB, DestBB);<br>
@@ -720,12 +793,14 @@ BasicBlock *FuncPGOInstrumentation<Edge,<br>
// Critical edges will be split.<br>
static void instrumentOneFunc(<br>
Function &F, Module *M, BranchProbabilityInfo *BPI, BlockFrequencyInfo *BFI,<br>
- std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers) {<br>
+ std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers,<br>
+ bool IsCS) {<br>
// Split indirectbr critical edges here before computing the MST rather than<br>
// later in getInstrBB() to avoid invalidating it.<br>
SplitIndirectBrCriticalEdges(F, BPI, BFI);<br>
+<br>
FuncPGOInstrumentation<PGOEdge, BBInfo> FuncInfo(F, ComdatMembers, true, BPI,<br>
- BFI);<br>
+ BFI, IsCS);<br>
unsigned NumCounters = FuncInfo.getNumCounters();<br>
<br>
uint32_t I = 0;<br>
@@ -852,10 +927,10 @@ public:<br>
PGOUseFunc(Function &Func, Module *Modu,<br>
std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers,<br>
BranchProbabilityInfo *BPI = nullptr,<br>
- BlockFrequencyInfo *BFIin = nullptr)<br>
+ BlockFrequencyInfo *BFIin = nullptr, bool IsCS = false)<br>
: F(Func), M(Modu), BFI(BFIin),<br>
- FuncInfo(Func, ComdatMembers, false, BPI, BFIin),<br>
- FreqAttr(FFA_Normal) {}<br>
+ FuncInfo(Func, ComdatMembers, false, BPI, BFIin, IsCS),<br>
+ FreqAttr(FFA_Normal), IsCS(IsCS) {}<br>
<br>
// Read counts for the instrumented BB from profile.<br>
bool readCounters(IndexedInstrProfReader *PGOReader, bool &AllZeros);<br>
@@ -928,6 +1003,9 @@ private:<br>
// Function hotness info derived from profile.<br>
FuncFreqAttr FreqAttr;<br>
<br>
+ // Is to use the context sensitive profile.<br>
+ bool IsCS;<br>
+<br>
// Find the Instrumented BB and set the value.<br>
void setInstrumentedCounts(const std::vector<uint64_t> &CountFromProfile);<br>
<br>
@@ -1021,23 +1099,31 @@ bool PGOUseFunc::readCounters(IndexedIns<br>
handleAllErrors(std::move(E), [&](const InstrProfError &IPE) {<br>
auto Err = IPE.get();<br>
bool SkipWarning = false;<br>
+ LLVM_DEBUG(dbgs() << "Error in reading profile for Func "<br>
+ << FuncInfo.FuncName << ": ");<br>
if (Err == instrprof_error::unknown_function) {<br>
- NumOfPGOMissing++;<br>
+ IsCS ? NumOfCSPGOMissing++ : NumOfPGOMissing++;<br>
SkipWarning = !PGOWarnMissing;<br>
+ LLVM_DEBUG(dbgs() << "unknown function");<br>
} else if (Err == instrprof_error::hash_mismatch ||<br>
Err == instrprof_error::malformed) {<br>
- NumOfPGOMismatch++;<br>
+ IsCS ? NumOfCSPGOMismatch++ : NumOfPGOMismatch++;<br>
SkipWarning =<br>
NoPGOWarnMismatch ||<br>
(NoPGOWarnMismatchComdat &&<br>
(F.hasComdat() ||<br>
F.getLinkage() == GlobalValue::AvailableExternallyLinkage));<br>
+ LLVM_DEBUG(dbgs() << "hash mismatch (skip=" << SkipWarning << ")");<br>
}<br>
<br>
+ LLVM_DEBUG(dbgs() << " IsCS=" << IsCS << "\n");<br>
if (SkipWarning)<br>
return;<br>
<br>
- std::string Msg = IPE.message() + std::string(" ") + F.getName().str();<br>
+ std::string Msg = IPE.message() + std::string(" ") + F.getName().str() +<br>
+ std::string(" Hash = ") +<br>
+ std::to_string(FuncInfo.FunctionHash);<br>
+<br>
Ctx.diagnose(<br>
DiagnosticInfoPGOProfile(M->getName().data(), Msg, DS_Warning));<br>
});<br>
@@ -1046,7 +1132,7 @@ bool PGOUseFunc::readCounters(IndexedIns<br>
ProfileRecord = std::move(Result.get());<br>
std::vector<uint64_t> &CountFromProfile = ProfileRecord.Counts;<br>
<br>
- NumOfPGOFunc++;<br>
+ IsCS ? NumOfCSPGOFunc++ : NumOfPGOFunc++;<br>
LLVM_DEBUG(dbgs() << CountFromProfile.size() << " counts\n");<br>
uint64_t ValueSum = 0;<br>
for (unsigned I = 0, S = CountFromProfile.size(); I < S; I++) {<br>
@@ -1061,7 +1147,11 @@ bool PGOUseFunc::readCounters(IndexedIns<br>
getBBInfo(nullptr).UnknownCountInEdge = 2;<br>
<br>
setInstrumentedCounts(CountFromProfile);<br>
+#if 0<br>
+ ProgramMaxCount = PGOReader->getMaximumFunctionCount(IsCS);<br>
+#else<br>
ProgramMaxCount = PGOReader->getMaximumFunctionCount();<br>
+#endif<br>
return true;<br>
}<br>
<br>
@@ -1166,7 +1256,8 @@ void PGOUseFunc::populateCounters() {<br>
// Assign the scaled count values to the BB with multiple out edges.<br>
void PGOUseFunc::setBranchWeights() {<br>
// Generate MD_prof metadata for every branch instruction.<br>
- LLVM_DEBUG(dbgs() << "\nSetting branch weights.\n");<br>
+ LLVM_DEBUG(dbgs() << "\nSetting branch weights for func " << F.getName()<br>
+ << " IsCS=" << IsCS << "\n");<br>
for (auto &BB : F) {<br>
Instruction *TI = BB.getTerminator();<br>
if (TI->getNumSuccessors() < 2)<br>
@@ -1174,6 +1265,7 @@ void PGOUseFunc::setBranchWeights() {<br>
if (!(isa<BranchInst>(TI) || isa<SwitchInst>(TI) ||<br>
isa<IndirectBrInst>(TI)))<br>
continue;<br>
+<br>
if (getBBInfo(&BB).CountValue == 0)<br>
continue;<br>
<br>
@@ -1351,24 +1443,6 @@ void PGOUseFunc::annotateValueSites(uint<br>
}<br>
}<br>
<br>
-// Create a COMDAT variable INSTR_PROF_RAW_VERSION_VAR to make the runtime<br>
-// aware this is an ir_level profile so it can set the version flag.<br>
-static void createIRLevelProfileFlagVariable(Module &M) {<br>
- Type *IntTy64 = Type::getInt64Ty(M.getContext());<br>
- uint64_t ProfileVersion = (INSTR_PROF_RAW_VERSION | VARIANT_MASK_IR_PROF);<br>
- auto IRLevelVersionVariable = new GlobalVariable(<br>
- M, IntTy64, true, GlobalVariable::ExternalLinkage,<br>
- Constant::getIntegerValue(IntTy64, APInt(64, ProfileVersion)),<br>
- INSTR_PROF_QUOTE(INSTR_PROF_RAW_VERSION_VAR));<br>
- IRLevelVersionVariable->setVisibility(GlobalValue::DefaultVisibility);<br>
- Triple TT(M.getTargetTriple());<br>
- if (!TT.supportsCOMDAT())<br>
- IRLevelVersionVariable->setLinkage(GlobalValue::WeakAnyLinkage);<br>
- else<br>
- IRLevelVersionVariable->setComdat(M.getOrInsertComdat(<br>
- StringRef(INSTR_PROF_QUOTE(INSTR_PROF_RAW_VERSION_VAR))));<br>
-}<br>
-<br>
// Collect the set of members for each Comdat in module M and store<br>
// in ComdatMembers.<br>
static void collectComdatMembers(<br>
@@ -1389,8 +1463,11 @@ static void collectComdatMembers(<br>
<br>
static bool InstrumentAllFunctions(<br>
Module &M, function_ref<BranchProbabilityInfo *(Function &)> LookupBPI,<br>
- function_ref<BlockFrequencyInfo *(Function &)> LookupBFI) {<br>
- createIRLevelProfileFlagVariable(M);<br>
+ function_ref<BlockFrequencyInfo *(Function &)> LookupBFI, bool IsCS) {<br>
+ // For the context-sensitve instrumentation, we should have a separated pass<br>
+ // (before LTO/ThinLTO linking) to create these variables.<br>
+ if (!IsCS)<br>
+ createIRLevelProfileFlagVar(M, /* IsCS */ false);<br>
std::unordered_multimap<Comdat *, GlobalValue *> ComdatMembers;<br>
collectComdatMembers(M, ComdatMembers);<br>
<br>
@@ -1399,7 +1476,7 @@ static bool InstrumentAllFunctions(<br>
continue;<br>
auto *BPI = LookupBPI(F);<br>
auto *BFI = LookupBFI(F);<br>
- instrumentOneFunc(F, &M, BPI, BFI, ComdatMembers);<br>
+ instrumentOneFunc(F, &M, BPI, BFI, ComdatMembers, IsCS);<br>
}<br>
return true;<br>
}<br>
@@ -1414,7 +1491,7 @@ bool PGOInstrumentationGenLegacyPass::ru<br>
auto LookupBFI = [this](Function &F) {<br>
return &this->getAnalysis<BlockFrequencyInfoWrapperPass>(F).getBFI();<br>
};<br>
- return InstrumentAllFunctions(M, LookupBPI, LookupBFI);<br>
+ return InstrumentAllFunctions(M, LookupBPI, LookupBFI, IsCS);<br>
}<br>
<br>
PreservedAnalyses PGOInstrumentationGen::run(Module &M,<br>
@@ -1428,7 +1505,7 @@ PreservedAnalyses PGOInstrumentationGen:<br>
return &FAM.getResult<BlockFrequencyAnalysis>(F);<br>
};<br>
<br>
- if (!InstrumentAllFunctions(M, LookupBPI, LookupBFI))<br>
+ if (!InstrumentAllFunctions(M, LookupBPI, LookupBFI, IsCS))<br>
return PreservedAnalyses::all();<br>
<br>
return PreservedAnalyses::none();<br>
@@ -1437,7 +1514,7 @@ PreservedAnalyses PGOInstrumentationGen:<br>
static bool annotateAllFunctions(<br>
Module &M, StringRef ProfileFileName, StringRef ProfileRemappingFileName,<br>
function_ref<BranchProbabilityInfo *(Function &)> LookupBPI,<br>
- function_ref<BlockFrequencyInfo *(Function &)> LookupBFI) {<br>
+ function_ref<BlockFrequencyInfo *(Function &)> LookupBFI, bool IsCS) {<br>
LLVM_DEBUG(dbgs() << "Read in profile counters: ");<br>
auto &Ctx = M.getContext();<br>
// Read the counter array from file.<br>
@@ -1458,6 +1535,11 @@ static bool annotateAllFunctions(<br>
StringRef("Cannot get PGOReader")));<br>
return false;<br>
}<br>
+#if 0<br>
+ if (!PGOReader->hasCSIRLevelProfile() && IsCS)<br>
+ return false;<br>
+#endif<br>
+<br>
// TODO: might need to change the warning once the clang option is finalized.<br>
if (!PGOReader->isIRLevelProfile()) {<br>
Ctx.diagnose(DiagnosticInfoPGOProfile(<br>
@@ -1477,7 +1559,7 @@ static bool annotateAllFunctions(<br>
// Split indirectbr critical edges here before computing the MST rather than<br>
// later in getInstrBB() to avoid invalidating it.<br>
SplitIndirectBrCriticalEdges(F, BPI, BFI);<br>
- PGOUseFunc Func(F, &M, ComdatMembers, BPI, BFI);<br>
+ PGOUseFunc Func(F, &M, ComdatMembers, BPI, BFI, IsCS);<br>
bool AllZeros = false;<br>
if (!Func.readCounters(PGOReader.get(), AllZeros))<br>
continue;<br>
@@ -1525,7 +1607,14 @@ static bool annotateAllFunctions(<br>
}<br>
}<br>
}<br>
+#if 0<br>
+ M.setProfileSummary(PGOReader->getSummary(IsCS).getMD(M.getContext()),<br>
+ IsCS ? ProfileSummary::PSK_CSInstr<br>
+ : ProfileSummary::PSK_Instr);<br>
+#else<br>
M.setProfileSummary(PGOReader->getSummary().getMD(M.getContext()));<br>
+#endif<br>
+<br>
// Set function hotness attribute from the profile.<br>
// We have to apply these attributes at the end because their presence<br>
// can affect the BranchProbabilityInfo of any callers, resulting in an<br>
@@ -1544,9 +1633,10 @@ static bool annotateAllFunctions(<br>
}<br>
<br>
PGOInstrumentationUse::PGOInstrumentationUse(std::string Filename,<br>
- std::string RemappingFilename)<br>
+ std::string RemappingFilename,<br>
+ bool IsCS)<br>
: ProfileFileName(std::move(Filename)),<br>
- ProfileRemappingFileName(std::move(RemappingFilename)) {<br>
+ ProfileRemappingFileName(std::move(RemappingFilename)), IsCS(IsCS) {<br>
if (!PGOTestProfileFile.empty())<br>
ProfileFileName = PGOTestProfileFile;<br>
if (!PGOTestProfileRemappingFile.empty())<br>
@@ -1566,7 +1656,7 @@ PreservedAnalyses PGOInstrumentationUse:<br>
};<br>
<br>
if (!annotateAllFunctions(M, ProfileFileName, ProfileRemappingFileName,<br>
- LookupBPI, LookupBFI))<br>
+ LookupBPI, LookupBFI, IsCS))<br>
return PreservedAnalyses::all();<br>
<br>
return PreservedAnalyses::none();<br>
@@ -1583,7 +1673,8 @@ bool PGOInstrumentationUseLegacyPass::ru<br>
return &this->getAnalysis<BlockFrequencyInfoWrapperPass>(F).getBFI();<br>
};<br>
<br>
- return annotateAllFunctions(M, ProfileFileName, "", LookupBPI, LookupBFI);<br>
+ return annotateAllFunctions(M, ProfileFileName, "", LookupBPI, LookupBFI,<br>
+ IsCS);<br>
}<br>
<br>
static std::string getSimpleNodeName(const BasicBlock *Node) {<br>
<br>
Modified: llvm/trunk/tools/gold/gold-plugin.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/tools/gold/gold-plugin.cpp?rev=354930&r1=354929&r2=354930&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/tools/gold/gold-plugin.cpp?rev=354930&r1=354929&r2=354930&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/tools/gold/gold-plugin.cpp (original)<br>
+++ llvm/trunk/tools/gold/gold-plugin.cpp Tue Feb 26 14:37:46 2019<br>
@@ -209,6 +209,10 @@ namespace options {<br>
static std::string OptRemarksFilename;<br>
static bool OptRemarksWithHotness = false;<br>
<br>
+ // Context sensitive PGO options.<br>
+ static std::string cs_profile_path;<br>
+ static bool cs_pgo_gen = false;<br>
+<br>
static void process_plugin_option(const char *opt_)<br>
{<br>
if (opt_ == nullptr)<br>
@@ -268,7 +272,11 @@ namespace options {<br>
} else if (opt == "disable-verify") {<br>
DisableVerify = true;<br>
} else if (opt.startswith("sample-profile=")) {<br>
- sample_profile= opt.substr(strlen("sample-profile="));<br>
+ sample_profile = opt.substr(strlen("sample-profile="));<br>
+ } else if (opt == "cs-profile-generate") {<br>
+ cs_pgo_gen = true;<br>
+ } else if (opt.startswith("cs-profile-path=")) {<br>
+ cs_profile_path = opt.substr(strlen("cs-profile-path="));<br>
} else if (opt == "new-pass-manager") {<br>
new_pass_manager = true;<br>
} else if (opt == "debug-pass-manager") {<br>
@@ -892,6 +900,10 @@ static std::unique_ptr<LTO> createLTO(In<br>
if (!options::sample_profile.empty())<br>
Conf.SampleProfile = options::sample_profile;<br>
<br>
+ if (!options::cs_profile_path.empty())<br>
+ Conf.CSIRProfile = options::cs_profile_path;<br>
+ Conf.RunCSIRInstr = options::cs_pgo_gen;<br>
+<br>
Conf.DwoDir = options::dwo_dir;<br>
<br>
// Set up optimization remarks handling.<br>
<br>
Modified: llvm/trunk/tools/llvm-lto2/llvm-lto2.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/tools/llvm-lto2/llvm-lto2.cpp?rev=354930&r1=354929&r2=354930&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/tools/llvm-lto2/llvm-lto2.cpp?rev=354930&r1=354929&r2=354930&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/tools/llvm-lto2/llvm-lto2.cpp (original)<br>
+++ llvm/trunk/tools/llvm-lto2/llvm-lto2.cpp Tue Feb 26 14:37:46 2019<br>
@@ -104,6 +104,15 @@ static cl::opt<std::string><br>
SamplePGOFile("lto-sample-profile-file",<br>
cl::desc("Specify a SamplePGO profile file"));<br>
<br>
+static cl::opt<std::string><br>
+ CSPGOFile("lto-cspgo-profile-file",<br>
+ cl::desc("Specify a context sensitive PGO profile file"));<br>
+<br>
+static cl::opt<bool><br>
+ RunCSIRInstr("lto-cspgo-gen",<br>
+ cl::desc("Run PGO context sensitive IR instrumentation"),<br>
+ cl::init(false), cl::Hidden);<br>
+<br>
static cl::opt<bool><br>
UseNewPM("use-new-pm",<br>
cl::desc("Run LTO passes using the new pass manager"),<br>
@@ -214,6 +223,8 @@ static int run(int argc, char **argv) {<br>
Conf.RemarksWithHotness = OptRemarksWithHotness;<br>
<br>
Conf.SampleProfile = SamplePGOFile;<br>
+ Conf.CSIRProfile = CSPGOFile;<br>
+ Conf.RunCSIRInstr = RunCSIRInstr;<br>
<br>
// Run a custom pipeline, if asked for.<br>
Conf.OptPipeline = OptPipeline;<br>
<br>
<br>
_______________________________________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br>
</blockquote></div>