[llvm] r354930 - [PGO] Context sensitive PGO (part 1)
Vlad Tsyrklevich via llvm-commits
llvm-commits at lists.llvm.org
Tue Feb 26 19:53:43 PST 2019
This change was causing MSan and UBSan failures
<http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/29937>
and
reverted in r354953.
On Tue, Feb 26, 2019 at 2:36 PM Rong Xu via llvm-commits <
llvm-commits at lists.llvm.org> wrote:
> Author: xur
> Date: Tue Feb 26 14:37:46 2019
> New Revision: 354930
>
> URL: http://llvm.org/viewvc/llvm-project?rev=354930&view=rev
> Log:
> [PGO] Context sensitive PGO (part 1)
>
> Current PGO profile counts are not context sensitive. The branch
> probabilities
> for the inlined functions are kept the same for all call-sites, and they
> might
> be very different from the actual branch probabilities. These suboptimal
> profiles can greatly affect some downstream optimizations, in particular
> for
> the machine basic block placement optimization.
>
> In this patch, we propose to have a post-inline PGO instrumentation/use
> pass,
> which we called Context Sensitive PGO (CSPGO). For the users who want the
> best
> possible performance, they can perform a second round of PGO
> instrument/use on
> the top of the regular PGO. They will have two sets of profile counts. The
> first pass profile will be manly for inline, indirect-call promotion, and
> CGSCC simplification pass optimizations. The second pass profile is for
> post-inline optimizations and code-gen optimizations.
>
> A typical usage:
> // Regular PGO instrumentation and generate pass1 profile.
> > clang -O2 -fprofile-generate source.c -o gen
> > ./gen
> > llvm-profdata merge default.*profraw -o pass1.profdata
> // CSPGO instrumentation.
> > clang -O2 -fprofile-use=pass1.profdata -fcs-profile-generate -o gen2
> > ./gen2
> // Merge two sets of profiles
> > llvm-profdata merge default.*profraw pass1.profdata -o profile.profdata
> // Use the combined profile. Pass manager will invoke two PGO use passes.
> > clang -O2 -fprofile-use=profile.profdata -o use
>
> This change touches many components in the compiler. The reviewed patch
> (D54175) will committed in phrases.
>
> Differential Revision: https://reviews.llvm.org/D54175
>
> Modified:
> llvm/trunk/include/llvm/InitializePasses.h
> llvm/trunk/include/llvm/LTO/Config.h
> llvm/trunk/include/llvm/LinkAllPasses.h
> llvm/trunk/include/llvm/ProfileData/InstrProf.h
> llvm/trunk/include/llvm/ProfileData/InstrProfData.inc
> llvm/trunk/include/llvm/Transforms/Instrumentation.h
> llvm/trunk/include/llvm/Transforms/Instrumentation/InstrProfiling.h
> llvm/trunk/include/llvm/Transforms/Instrumentation/PGOInstrumentation.h
> llvm/trunk/lib/Passes/PassBuilder.cpp
> llvm/trunk/lib/ProfileData/InstrProf.cpp
> llvm/trunk/lib/Transforms/Instrumentation/InstrProfiling.cpp
> llvm/trunk/lib/Transforms/Instrumentation/PGOInstrumentation.cpp
> llvm/trunk/tools/gold/gold-plugin.cpp
> llvm/trunk/tools/llvm-lto2/llvm-lto2.cpp
>
> Modified: llvm/trunk/include/llvm/InitializePasses.h
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/InitializePasses.h?rev=354930&r1=354929&r2=354930&view=diff
>
> ==============================================================================
> --- llvm/trunk/include/llvm/InitializePasses.h (original)
> +++ llvm/trunk/include/llvm/InitializePasses.h Tue Feb 26 14:37:46 2019
> @@ -299,6 +299,7 @@ void initializePEIPass(PassRegistry&);
> void initializePGOIndirectCallPromotionLegacyPassPass(PassRegistry&);
> void initializePGOInstrumentationGenLegacyPassPass(PassRegistry&);
> void initializePGOInstrumentationUseLegacyPassPass(PassRegistry&);
> +void
> initializePGOInstrumentationGenCreateVarLegacyPassPass(PassRegistry&);
> void initializePGOMemOPSizeOptLegacyPassPass(PassRegistry&);
> void initializePHIEliminationPass(PassRegistry&);
> void initializePartialInlinerLegacyPassPass(PassRegistry&);
>
> Modified: llvm/trunk/include/llvm/LTO/Config.h
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/LTO/Config.h?rev=354930&r1=354929&r2=354930&view=diff
>
> ==============================================================================
> --- llvm/trunk/include/llvm/LTO/Config.h (original)
> +++ llvm/trunk/include/llvm/LTO/Config.h Tue Feb 26 14:37:46 2019
> @@ -55,6 +55,9 @@ struct Config {
> /// Disable entirely the optimizer, including importing for ThinLTO
> bool CodeGenOnly = false;
>
> + /// Run PGO context sensitive IR instrumentation.
> + bool RunCSIRInstr = false;
> +
> /// If this field is set, the set of passes run in the middle-end
> optimizer
> /// will be the one specified by the string. Only works with the new
> pass
> /// manager as the old one doesn't have this ability.
> @@ -73,6 +76,9 @@ struct Config {
> /// with this triple.
> std::string DefaultTriple;
>
> + /// Context Sensitive PGO profile path.
> + std::string CSIRProfile;
> +
> /// Sample PGO profile path.
> std::string SampleProfile;
>
>
> Modified: llvm/trunk/include/llvm/LinkAllPasses.h
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/LinkAllPasses.h?rev=354930&r1=354929&r2=354930&view=diff
>
> ==============================================================================
> --- llvm/trunk/include/llvm/LinkAllPasses.h (original)
> +++ llvm/trunk/include/llvm/LinkAllPasses.h Tue Feb 26 14:37:46 2019
> @@ -102,6 +102,7 @@ namespace {
> (void) llvm::createGCOVProfilerPass();
> (void) llvm::createPGOInstrumentationGenLegacyPass();
> (void) llvm::createPGOInstrumentationUseLegacyPass();
> + (void) llvm::createPGOInstrumentationGenCreateVarLegacyPass();
> (void) llvm::createPGOIndirectCallPromotionLegacyPass();
> (void) llvm::createPGOMemOPSizeOptLegacyPass();
> (void) llvm::createInstrProfilingLegacyPass();
>
> Modified: llvm/trunk/include/llvm/ProfileData/InstrProf.h
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/ProfileData/InstrProf.h?rev=354930&r1=354929&r2=354930&view=diff
>
> ==============================================================================
> --- llvm/trunk/include/llvm/ProfileData/InstrProf.h (original)
> +++ llvm/trunk/include/llvm/ProfileData/InstrProf.h Tue Feb 26 14:37:46
> 2019
> @@ -767,10 +767,20 @@ struct NamedInstrProfRecord : InstrProfR
> StringRef Name;
> uint64_t Hash;
>
> + // We reserve this bit as the flag for context sensitive profile record.
> + static const int CS_FLAG_IN_FUNC_HASH = 60;
> +
> NamedInstrProfRecord() = default;
> NamedInstrProfRecord(StringRef Name, uint64_t Hash,
> std::vector<uint64_t> Counts)
> : InstrProfRecord(std::move(Counts)), Name(Name), Hash(Hash) {}
> +
> + static bool hasCSFlagInHash(uint64_t FuncHash) {
> + return ((FuncHash >> CS_FLAG_IN_FUNC_HASH) & 1);
> + }
> + static void setCSFlagInHash(uint64_t &FuncHash) {
> + FuncHash |= ((uint64_t)1 << CS_FLAG_IN_FUNC_HASH);
> + }
> };
>
> uint32_t InstrProfRecord::getNumValueKinds() const {
> @@ -1004,6 +1014,8 @@ namespace RawInstrProf {
> // from control data struct is changed from raw pointer to Name's MD5
> value.
> // Version 4: ValueDataBegin and ValueDataSizes fields are removed from
> the
> // raw header.
> +// Version 5: Bit 60 of FuncHash is reserved for the flag for the context
> +// sensitive records.
> const uint64_t Version = INSTR_PROF_RAW_VERSION;
>
> template <class IntPtrT> inline uint64_t getMagic();
> @@ -1040,6 +1052,10 @@ struct Header {
> void getMemOPSizeRangeFromOption(StringRef Str, int64_t &RangeStart,
> int64_t &RangeLast);
>
> +// Create a COMDAT variable INSTR_PROF_RAW_VERSION_VAR to make the runtime
> +// aware this is an ir_level profile so it can set the version flag.
> +void createIRLevelProfileFlagVar(Module &M, bool IsCS);
> +
> // Create the variable for the profile file name.
> void createProfileFileNameVar(Module &M, StringRef InstrProfileOutput);
>
>
> Modified: llvm/trunk/include/llvm/ProfileData/InstrProfData.inc
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/ProfileData/InstrProfData.inc?rev=354930&r1=354929&r2=354930&view=diff
>
> ==============================================================================
> --- llvm/trunk/include/llvm/ProfileData/InstrProfData.inc (original)
> +++ llvm/trunk/include/llvm/ProfileData/InstrProfData.inc Tue Feb 26
> 14:37:46 2019
> @@ -635,10 +635,12 @@ serializeValueProfDataFrom(ValueProfReco
> * version for other variants of profile. We set the lowest bit of the
> upper 8
> * bits (i.e. bit 56) to 1 to indicate if this is an IR-level
> instrumentaiton
> * generated profile, and 0 if this is a Clang FE generated profile.
> + * 1 in bit 57 indicates there are context-sensitive records in the
> profile.
> */
> #define VARIANT_MASKS_ALL 0xff00000000000000ULL
> #define GET_VERSION(V) ((V) & ~VARIANT_MASKS_ALL)
> #define VARIANT_MASK_IR_PROF (0x1ULL << 56)
> +#define VARIANT_MASK_CSIR_PROF (0x1ULL << 57)
> #define INSTR_PROF_RAW_VERSION_VAR __llvm_profile_raw_version
> #define INSTR_PROF_PROFILE_RUNTIME_VAR __llvm_profile_runtime
>
>
> Modified: llvm/trunk/include/llvm/Transforms/Instrumentation.h
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Transforms/Instrumentation.h?rev=354930&r1=354929&r2=354930&view=diff
>
> ==============================================================================
> --- llvm/trunk/include/llvm/Transforms/Instrumentation.h (original)
> +++ llvm/trunk/include/llvm/Transforms/Instrumentation.h Tue Feb 26
> 14:37:46 2019
> @@ -87,10 +87,14 @@ struct GCOVOptions {
> ModulePass *createGCOVProfilerPass(const GCOVOptions &Options =
> GCOVOptions::getDefault());
>
> -// PGO Instrumention
> -ModulePass *createPGOInstrumentationGenLegacyPass();
> +// PGO Instrumention. Parameter IsCS indicates if this is the context
> senstive
> +// instrumentation.
> +ModulePass *createPGOInstrumentationGenLegacyPass(bool IsCS = false);
> ModulePass *
> -createPGOInstrumentationUseLegacyPass(StringRef Filename = StringRef(""));
> +createPGOInstrumentationUseLegacyPass(StringRef Filename = StringRef(""),
> + bool IsCS = false);
> +ModulePass *createPGOInstrumentationGenCreateVarLegacyPass(
> + StringRef CSInstrName = StringRef(""));
> ModulePass *createPGOIndirectCallPromotionLegacyPass(bool InLTO = false,
> bool SamplePGO =
> false);
> FunctionPass *createPGOMemOPSizeOptLegacyPass();
> @@ -132,15 +136,19 @@ struct InstrProfOptions {
> // Use atomic profile counter increments.
> bool Atomic = false;
>
> + // Use BFI to guide register promotion
> + bool UseBFIInPromotion = false;
> +
> // Name of the profile file to use as output
> std::string InstrProfileOutput;
>
> InstrProfOptions() = default;
> };
>
> -/// Insert frontend instrumentation based profiling.
> +/// Insert frontend instrumentation based profiling. Parameter IsCS
> indicates if
> +// this is the context senstive instrumentation.
> ModulePass *createInstrProfilingLegacyPass(
> - const InstrProfOptions &Options = InstrProfOptions());
> + const InstrProfOptions &Options = InstrProfOptions(), bool IsCS =
> false);
>
> FunctionPass *createHWAddressSanitizerPass(bool CompileKernel = false,
> bool Recover = false);
>
> Modified:
> llvm/trunk/include/llvm/Transforms/Instrumentation/InstrProfiling.h
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Transforms/Instrumentation/InstrProfiling.h?rev=354930&r1=354929&r2=354930&view=diff
>
> ==============================================================================
> --- llvm/trunk/include/llvm/Transforms/Instrumentation/InstrProfiling.h
> (original)
> +++ llvm/trunk/include/llvm/Transforms/Instrumentation/InstrProfiling.h
> Tue Feb 26 14:37:46 2019
> @@ -35,7 +35,8 @@ using LoadStorePair = std::pair<Instruct
> class InstrProfiling : public PassInfoMixin<InstrProfiling> {
> public:
> InstrProfiling() = default;
> - InstrProfiling(const InstrProfOptions &Options) : Options(Options) {}
> + InstrProfiling(const InstrProfOptions &Options, bool IsCS)
> + : Options(Options), IsCS(IsCS) {}
>
> PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM);
> bool run(Module &M, const TargetLibraryInfo &TLI);
> @@ -60,6 +61,9 @@ private:
> GlobalVariable *NamesVar;
> size_t NamesSize;
>
> + // Is this lowering for the context-sensitive instrumentation.
> + bool IsCS;
> +
> // vector of counter load/store pairs to be register promoted.
> std::vector<LoadStorePair> PromotionCandidates;
>
>
> Modified:
> llvm/trunk/include/llvm/Transforms/Instrumentation/PGOInstrumentation.h
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Transforms/Instrumentation/PGOInstrumentation.h?rev=354930&r1=354929&r2=354930&view=diff
>
> ==============================================================================
> ---
> llvm/trunk/include/llvm/Transforms/Instrumentation/PGOInstrumentation.h
> (original)
> +++
> llvm/trunk/include/llvm/Transforms/Instrumentation/PGOInstrumentation.h Tue
> Feb 26 14:37:46 2019
> @@ -17,6 +17,7 @@
>
> #include "llvm/ADT/ArrayRef.h"
> #include "llvm/IR/PassManager.h"
> +#include "llvm/ProfileData/InstrProf.h"
> #include <cstdint>
> #include <string>
>
> @@ -27,22 +28,50 @@ class Instruction;
> class Module;
>
> /// The instrumentation (profile-instr-gen) pass for IR based PGO.
> +// We use this pass to create COMDAT profile variables for context
> +// sensitive PGO (CSPGO). The reason to have a pass for this is CSPGO
> +// can be run after LTO/ThinLTO linking. Lld linker needs to see
> +// all the COMDAT variables before linking. So we have this pass
> +// always run before linking for CSPGO.
> +class PGOInstrumentationGenCreateVar
> + : public PassInfoMixin<PGOInstrumentationGenCreateVar> {
> +public:
> + PGOInstrumentationGenCreateVar(std::string CSInstrName = "")
> + : CSInstrName(CSInstrName) {}
> + PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM) {
> + createProfileFileNameVar(M, CSInstrName);
> + createIRLevelProfileFlagVar(M, /* IsCS */ true);
> + return PreservedAnalyses::all();
> + }
> +
> +private:
> + std::string CSInstrName;
> +};
> +
> +/// The instrumentation (profile-instr-gen) pass for IR based PGO.
> class PGOInstrumentationGen : public PassInfoMixin<PGOInstrumentationGen>
> {
> public:
> + PGOInstrumentationGen(bool IsCS = false) : IsCS(IsCS) {}
> PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM);
> +
> +private:
> + // If this is a context sensitive instrumentation.
> + bool IsCS;
> };
>
> /// The profile annotation (profile-instr-use) pass for IR based PGO.
> class PGOInstrumentationUse : public PassInfoMixin<PGOInstrumentationUse>
> {
> public:
> PGOInstrumentationUse(std::string Filename = "",
> - std::string RemappingFilename = "");
> + std::string RemappingFilename = "", bool IsCS =
> false);
>
> PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM);
>
> private:
> std::string ProfileFileName;
> std::string ProfileRemappingFileName;
> + // If this is a context sensitive instrumentation.
> + bool IsCS;
> };
>
> /// The indirect function call promotion pass.
>
> Modified: llvm/trunk/lib/Passes/PassBuilder.cpp
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Passes/PassBuilder.cpp?rev=354930&r1=354929&r2=354930&view=diff
>
> ==============================================================================
> --- llvm/trunk/lib/Passes/PassBuilder.cpp (original)
> +++ llvm/trunk/lib/Passes/PassBuilder.cpp Tue Feb 26 14:37:46 2019
> @@ -569,7 +569,8 @@ void PassBuilder::addPGOInstrPasses(Modu
> if (!ProfileGenFile.empty())
> Options.InstrProfileOutput = ProfileGenFile;
> Options.DoCounterPromotion = true;
> - MPM.addPass(InstrProfiling(Options));
> + Options.UseBFIInPromotion = false;
> + MPM.addPass(InstrProfiling(Options, false));
> }
>
> if (!ProfileUseFile.empty())
>
> Modified: llvm/trunk/lib/ProfileData/InstrProf.cpp
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/ProfileData/InstrProf.cpp?rev=354930&r1=354929&r2=354930&view=diff
>
> ==============================================================================
> --- llvm/trunk/lib/ProfileData/InstrProf.cpp (original)
> +++ llvm/trunk/lib/ProfileData/InstrProf.cpp Tue Feb 26 14:37:46 2019
> @@ -1011,6 +1011,25 @@ void getMemOPSizeRangeFromOption(StringR
> assert(RangeLast >= RangeStart);
> }
>
> +// Create a COMDAT variable INSTR_PROF_RAW_VERSION_VAR to make the runtime
> +// aware this is an ir_level profile so it can set the version flag.
> +void createIRLevelProfileFlagVar(Module &M, bool IsCS) {
> + const StringRef VarName(INSTR_PROF_QUOTE(INSTR_PROF_RAW_VERSION_VAR));
> + Type *IntTy64 = Type::getInt64Ty(M.getContext());
> + uint64_t ProfileVersion = (INSTR_PROF_RAW_VERSION |
> VARIANT_MASK_IR_PROF);
> + if (IsCS)
> + ProfileVersion |= VARIANT_MASK_CSIR_PROF;
> + auto IRLevelVersionVariable = new GlobalVariable(
> + M, IntTy64, true, GlobalValue::WeakAnyLinkage,
> + Constant::getIntegerValue(IntTy64, APInt(64, ProfileVersion)),
> VarName);
> + IRLevelVersionVariable->setVisibility(GlobalValue::DefaultVisibility);
> + Triple TT(M.getTargetTriple());
> + if (TT.supportsCOMDAT()) {
> + IRLevelVersionVariable->setLinkage(GlobalValue::ExternalLinkage);
> + IRLevelVersionVariable->setComdat(M.getOrInsertComdat(VarName));
> + }
> +}
> +
> // Create the variable for the profile file name.
> void createProfileFileNameVar(Module &M, StringRef InstrProfileOutput) {
> if (InstrProfileOutput.empty())
>
> Modified: llvm/trunk/lib/Transforms/Instrumentation/InstrProfiling.cpp
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Instrumentation/InstrProfiling.cpp?rev=354930&r1=354929&r2=354930&view=diff
>
> ==============================================================================
> --- llvm/trunk/lib/Transforms/Instrumentation/InstrProfiling.cpp (original)
> +++ llvm/trunk/lib/Transforms/Instrumentation/InstrProfiling.cpp Tue Feb
> 26 14:37:46 2019
> @@ -18,6 +18,8 @@
> #include "llvm/ADT/StringRef.h"
> #include "llvm/ADT/Triple.h"
> #include "llvm/ADT/Twine.h"
> +#include "llvm/Analysis/BlockFrequencyInfo.h"
> +#include "llvm/Analysis/BranchProbabilityInfo.h"
> #include "llvm/Analysis/LoopInfo.h"
> #include "llvm/Analysis/TargetLibraryInfo.h"
> #include "llvm/IR/Attributes.h"
> @@ -147,8 +149,8 @@ public:
> static char ID;
>
> InstrProfilingLegacyPass() : ModulePass(ID) {}
> - InstrProfilingLegacyPass(const InstrProfOptions &Options)
> - : ModulePass(ID), InstrProf(Options) {}
> + InstrProfilingLegacyPass(const InstrProfOptions &Options, bool IsCS)
> + : ModulePass(ID), InstrProf(Options, IsCS) {}
>
> StringRef getPassName() const override {
> return "Frontend instrumentation-based coverage lowering";
> @@ -232,9 +234,9 @@ class PGOCounterPromoter {
> public:
> PGOCounterPromoter(
> DenseMap<Loop *, SmallVector<LoadStorePair, 8>> &LoopToCands,
> - Loop &CurLoop, LoopInfo &LI)
> + Loop &CurLoop, LoopInfo &LI, BlockFrequencyInfo *BFI)
> : LoopToCandidates(LoopToCands), ExitBlocks(), InsertPts(),
> L(CurLoop),
> - LI(LI) {
> + LI(LI), BFI(BFI) {
>
> SmallVector<BasicBlock *, 8> LoopExitBlocks;
> SmallPtrSet<BasicBlock *, 8> BlockSet;
> @@ -263,6 +265,20 @@ public:
> SSAUpdater SSA(&NewPHIs);
> Value *InitVal = ConstantInt::get(Cand.first->getType(), 0);
>
> + // If BFI is set, we will use it to guide the promotions.
> + if (BFI) {
> + auto *BB = Cand.first->getParent();
> + auto InstrCount = BFI->getBlockProfileCount(BB);
> + if (!InstrCount)
> + continue;
> + auto PreheaderCount =
> BFI->getBlockProfileCount(L.getLoopPreheader());
> + // If the average loop trip count is not greater than 1.5, we skip
> + // promotion.
> + if (PreheaderCount &&
> + (PreheaderCount.getValue() * 3) >= (InstrCount.getValue() *
> 2))
> + continue;
> + }
> +
> PGOCounterPromoterHelper Promoter(Cand.first, Cand.second, SSA,
> InitVal,
> L.getLoopPreheader(), ExitBlocks,
> InsertPts, LoopToCandidates, LI);
> @@ -312,6 +328,11 @@ private:
>
> SmallVector<BasicBlock *, 8> ExitingBlocks;
> LP->getExitingBlocks(ExitingBlocks);
> +
> + // If BFI is set, we do more aggressive promotions based on BFI.
> + if (BFI)
> + return (unsigned)-1;
> +
> // Not considierered speculative.
> if (ExitingBlocks.size() == 1)
> return MaxNumOfPromotionsPerLoop;
> @@ -343,6 +364,7 @@ private:
> SmallVector<Instruction *, 8> InsertPts;
> Loop &L;
> LoopInfo &LI;
> + BlockFrequencyInfo *BFI;
> };
>
> } // end anonymous namespace
> @@ -365,8 +387,9 @@ INITIALIZE_PASS_END(
> "Frontend instrumentation-based coverage lowering.", false, false)
>
> ModulePass *
> -llvm::createInstrProfilingLegacyPass(const InstrProfOptions &Options) {
> - return new InstrProfilingLegacyPass(Options);
> +llvm::createInstrProfilingLegacyPass(const InstrProfOptions &Options,
> + bool IsCS) {
> + return new InstrProfilingLegacyPass(Options, IsCS);
> }
>
> static InstrProfIncrementInst *castToIncrementInst(Instruction *Instr) {
> @@ -415,6 +438,13 @@ void InstrProfiling::promoteCounterLoadS
> LoopInfo LI(DT);
> DenseMap<Loop *, SmallVector<LoadStorePair, 8>> LoopPromotionCandidates;
>
> + std::unique_ptr<BlockFrequencyInfo> BFI;
> + if (Options.UseBFIInPromotion) {
> + std::unique_ptr<BranchProbabilityInfo> BPI;
> + BPI.reset(new BranchProbabilityInfo(*F, LI, TLI));
> + BFI.reset(new BlockFrequencyInfo(*F, *BPI, LI));
> + }
> +
> for (const auto &LoadStore : PromotionCandidates) {
> auto *CounterLoad = LoadStore.first;
> auto *CounterStore = LoadStore.second;
> @@ -430,7 +460,7 @@ void InstrProfiling::promoteCounterLoadS
> // Do a post-order traversal of the loops so that counter updates can be
> // iteratively hoisted outside the loop nest.
> for (auto *Loop : llvm::reverse(Loops)) {
> - PGOCounterPromoter Promoter(LoopPromotionCandidates, *Loop, LI);
> + PGOCounterPromoter Promoter(LoopPromotionCandidates, *Loop, LI,
> BFI.get());
> Promoter.run(&TotalCountersPromoted);
> }
> }
> @@ -681,7 +711,6 @@ static bool needsRuntimeRegistrationOfSe
> // Don't do this for Darwin. compiler-rt uses linker magic.
> if (TT.isOSDarwin())
> return false;
> -
> // Use linker script magic to get data/cnts/name start/end.
> if (TT.isOSLinux() || TT.isOSFreeBSD() || TT.isOSNetBSD() ||
> TT.isOSFuchsia() || TT.isPS4CPU() || TT.isOSWindows())
> @@ -985,8 +1014,12 @@ void InstrProfiling::emitUses() {
> }
>
> void InstrProfiling::emitInitialization() {
> - // Create variable for profile name.
> - createProfileFileNameVar(*M, Options.InstrProfileOutput);
> + // Create ProfileFileName variable. Don't don't this for the
> + // context-sensitive instrumentation lowering: This lowering is after
> + // LTO/ThinLTO linking. Pass PGOInstrumentationGenCreateVar should
> + // have already create the variable before LTO/ThinLTO linking.
> + if (!IsCS)
> + createProfileFileNameVar(*M, Options.InstrProfileOutput);
> Function *RegisterF = M->getFunction(getInstrProfRegFuncsName());
> if (!RegisterF)
> return;
>
> Modified: llvm/trunk/lib/Transforms/Instrumentation/PGOInstrumentation.cpp
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Instrumentation/PGOInstrumentation.cpp?rev=354930&r1=354929&r2=354930&view=diff
>
> ==============================================================================
> --- llvm/trunk/lib/Transforms/Instrumentation/PGOInstrumentation.cpp
> (original)
> +++ llvm/trunk/lib/Transforms/Instrumentation/PGOInstrumentation.cpp Tue
> Feb 26 14:37:46 2019
> @@ -65,6 +65,7 @@
> #include "llvm/Analysis/IndirectCallVisitor.h"
> #include "llvm/Analysis/LoopInfo.h"
> #include "llvm/Analysis/OptimizationRemarkEmitter.h"
> +#include "llvm/Analysis/ProfileSummaryInfo.h"
> #include "llvm/IR/Attributes.h"
> #include "llvm/IR/BasicBlock.h"
> #include "llvm/IR/CFG.h"
> @@ -132,6 +133,19 @@ STATISTIC(NumOfPGOFunc, "Number of funct
> STATISTIC(NumOfPGOMismatch, "Number of functions having mismatch
> profile.");
> STATISTIC(NumOfPGOMissing, "Number of functions without profile.");
> STATISTIC(NumOfPGOICall, "Number of indirect call value
> instrumentations.");
> +STATISTIC(NumOfCSPGOInstrument, "Number of edges instrumented in CSPGO.");
> +STATISTIC(NumOfCSPGOSelectInsts,
> + "Number of select instruction instrumented in CSPGO.");
> +STATISTIC(NumOfCSPGOMemIntrinsics,
> + "Number of mem intrinsics instrumented in CSPGO.");
> +STATISTIC(NumOfCSPGOEdge, "Number of edges in CSPGO.");
> +STATISTIC(NumOfCSPGOBB, "Number of basic-blocks in CSPGO.");
> +STATISTIC(NumOfCSPGOSplit, "Number of critical edge splits in CSPGO.");
> +STATISTIC(NumOfCSPGOFunc,
> + "Number of functions having valid profile counts in CSPGO.");
> +STATISTIC(NumOfCSPGOMismatch,
> + "Number of functions having mismatch profile in CSPGO.");
> +STATISTIC(NumOfCSPGOMissing, "Number of functions without profile in
> CSPGO.");
>
> // Command line option to specify the file to read profile from. This is
> // mainly used for testing.
> @@ -383,7 +397,8 @@ class PGOInstrumentationGenLegacyPass :
> public:
> static char ID;
>
> - PGOInstrumentationGenLegacyPass() : ModulePass(ID) {
> + PGOInstrumentationGenLegacyPass(bool IsCS = false)
> + : ModulePass(ID), IsCS(IsCS) {
> initializePGOInstrumentationGenLegacyPassPass(
> *PassRegistry::getPassRegistry());
> }
> @@ -391,6 +406,8 @@ public:
> StringRef getPassName() const override { return
> "PGOInstrumentationGenPass"; }
>
> private:
> + // Is this is context-sensitive instrumentation.
> + bool IsCS;
> bool runOnModule(Module &M) override;
>
> void getAnalysisUsage(AnalysisUsage &AU) const override {
> @@ -403,8 +420,8 @@ public:
> static char ID;
>
> // Provide the profile filename as the parameter.
> - PGOInstrumentationUseLegacyPass(std::string Filename = "")
> - : ModulePass(ID), ProfileFileName(std::move(Filename)) {
> + PGOInstrumentationUseLegacyPass(std::string Filename = "", bool IsCS =
> false)
> + : ModulePass(ID), ProfileFileName(std::move(Filename)), IsCS(IsCS) {
> if (!PGOTestProfileFile.empty())
> ProfileFileName = PGOTestProfileFile;
> initializePGOInstrumentationUseLegacyPassPass(
> @@ -415,14 +432,38 @@ public:
>
> private:
> std::string ProfileFileName;
> + // Is this is context-sensitive instrumentation use.
> + bool IsCS;
>
> bool runOnModule(Module &M) override;
>
> void getAnalysisUsage(AnalysisUsage &AU) const override {
> + AU.addRequired<ProfileSummaryInfoWrapperPass>();
> AU.addRequired<BlockFrequencyInfoWrapperPass>();
> }
> };
>
> +class PGOInstrumentationGenCreateVarLegacyPass : public ModulePass {
> +public:
> + static char ID;
> + StringRef getPassName() const override {
> + return "PGOInstrumentationGenCreateVarPass";
> + }
> + PGOInstrumentationGenCreateVarLegacyPass(std::string CSInstrName = "")
> + : ModulePass(ID), InstrProfileOutput(CSInstrName) {
> + initializePGOInstrumentationGenCreateVarLegacyPassPass(
> + *PassRegistry::getPassRegistry());
> + }
> +
> +private:
> + bool runOnModule(Module &M) override {
> + createProfileFileNameVar(M, InstrProfileOutput);
> + createIRLevelProfileFlagVar(M, true);
> + return false;
> + }
> + std::string InstrProfileOutput;
> +};
> +
> } // end anonymous namespace
>
> char PGOInstrumentationGenLegacyPass::ID = 0;
> @@ -434,8 +475,8 @@ INITIALIZE_PASS_DEPENDENCY(BranchProbabi
> INITIALIZE_PASS_END(PGOInstrumentationGenLegacyPass, "pgo-instr-gen",
> "PGO instrumentation.", false, false)
>
> -ModulePass *llvm::createPGOInstrumentationGenLegacyPass() {
> - return new PGOInstrumentationGenLegacyPass();
> +ModulePass *llvm::createPGOInstrumentationGenLegacyPass(bool IsCS) {
> + return new PGOInstrumentationGenLegacyPass(IsCS);
> }
>
> char PGOInstrumentationUseLegacyPass::ID = 0;
> @@ -444,11 +485,25 @@ INITIALIZE_PASS_BEGIN(PGOInstrumentation
> "Read PGO instrumentation profile.", false, false)
> INITIALIZE_PASS_DEPENDENCY(BlockFrequencyInfoWrapperPass)
> INITIALIZE_PASS_DEPENDENCY(BranchProbabilityInfoWrapperPass)
> +INITIALIZE_PASS_DEPENDENCY(ProfileSummaryInfoWrapperPass)
> INITIALIZE_PASS_END(PGOInstrumentationUseLegacyPass, "pgo-instr-use",
> "Read PGO instrumentation profile.", false, false)
>
> -ModulePass *llvm::createPGOInstrumentationUseLegacyPass(StringRef
> Filename) {
> - return new PGOInstrumentationUseLegacyPass(Filename.str());
> +ModulePass *llvm::createPGOInstrumentationUseLegacyPass(StringRef
> Filename,
> + bool IsCS) {
> + return new PGOInstrumentationUseLegacyPass(Filename.str(), IsCS);
> +}
> +
> +char PGOInstrumentationGenCreateVarLegacyPass::ID = 0;
> +
> +INITIALIZE_PASS(PGOInstrumentationGenCreateVarLegacyPass,
> + "pgo-instr-gen-create-var",
> + "Create PGO instrumentation version variable for CSPGO.",
> false,
> + false)
> +
> +ModulePass *
> +llvm::createPGOInstrumentationGenCreateVarLegacyPass(StringRef
> CSInstrName) {
> + return new PGOInstrumentationGenCreateVarLegacyPass(CSInstrName);
> }
>
> namespace {
> @@ -496,6 +551,9 @@ template <class Edge, class BBInfo> clas
> private:
> Function &F;
>
> + // Is this is context-sensitive instrumentation.
> + bool IsCS;
> +
> // A map that stores the Comdat group in function F.
> std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers;
>
> @@ -535,15 +593,23 @@ public:
> Function &Func,
> std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers,
> bool CreateGlobalVar = false, BranchProbabilityInfo *BPI = nullptr,
> - BlockFrequencyInfo *BFI = nullptr)
> - : F(Func), ComdatMembers(ComdatMembers), ValueSites(IPVK_Last + 1),
> - SIVisitor(Func), MIVisitor(Func), MST(F, BPI, BFI) {
> + BlockFrequencyInfo *BFI = nullptr, bool IsCS = false)
> + : F(Func), IsCS(IsCS), ComdatMembers(ComdatMembers),
> + ValueSites(IPVK_Last + 1), SIVisitor(Func), MIVisitor(Func),
> + MST(F, BPI, BFI) {
> // This should be done before CFG hash computation.
> SIVisitor.countSelects(Func);
> MIVisitor.countMemIntrinsics(Func);
> - NumOfPGOSelectInsts += SIVisitor.getNumOfSelectInsts();
> - NumOfPGOMemIntrinsics += MIVisitor.getNumOfMemIntrinsics();
> - ValueSites[IPVK_IndirectCallTarget] = findIndirectCalls(Func);
> + if (!IsCS) {
> + NumOfPGOSelectInsts += SIVisitor.getNumOfSelectInsts();
> + NumOfPGOMemIntrinsics += MIVisitor.getNumOfMemIntrinsics();
> + NumOfPGOBB += MST.BBInfos.size();
> + ValueSites[IPVK_IndirectCallTarget] = findIndirectCalls(Func);
> + } else {
> + NumOfCSPGOSelectInsts += SIVisitor.getNumOfSelectInsts();
> + NumOfCSPGOMemIntrinsics += MIVisitor.getNumOfMemIntrinsics();
> + NumOfCSPGOBB += MST.BBInfos.size();
> + }
> ValueSites[IPVK_MemOPSize] = MIVisitor.findMemIntrinsics(Func);
>
> FuncName = getPGOFuncName(F);
> @@ -552,13 +618,12 @@ public:
> renameComdatFunction();
> LLVM_DEBUG(dumpInfo("after CFGMST"));
>
> - NumOfPGOBB += MST.BBInfos.size();
> for (auto &E : MST.AllEdges) {
> if (E->Removed)
> continue;
> - NumOfPGOEdge++;
> + IsCS ? NumOfCSPGOEdge++ : NumOfPGOEdge++;
> if (!E->InMST)
> - NumOfPGOInstrument++;
> + IsCS ? NumOfCSPGOInstrument++ : NumOfPGOInstrument++;
> }
>
> if (CreateGlobalVar)
> @@ -597,9 +662,17 @@ void FuncPGOInstrumentation<Edge, BBInfo
> }
> }
> JC.update(Indexes);
> +
> + // Hash format for context sensitive profile. Reserve 4 bits for other
> + // information.
> FunctionHash = (uint64_t)SIVisitor.getNumOfSelectInsts() << 56 |
> (uint64_t)ValueSites[IPVK_IndirectCallTarget].size() <<
> 48 |
> + //(uint64_t)ValueSites[IPVK_MemOPSize].size() << 40 |
> (uint64_t)MST.AllEdges.size() << 32 | JC.getCRC();
> + // Reserve bit 60-63 for other information purpose.
> + FunctionHash &= 0x0FFFFFFFFFFFFFFF;
> + if (IsCS)
> + NamedInstrProfRecord::setCSFlagInHash(FunctionHash);
> LLVM_DEBUG(dbgs() << "Function Hash Computation for " << F.getName() <<
> ":\n"
> << " CRC = " << JC.getCRC()
> << ", Selects = " << SIVisitor.getNumOfSelectInsts()
> @@ -705,7 +778,7 @@ BasicBlock *FuncPGOInstrumentation<Edge,
>
> // For a critical edge, we have to split. Instrument the newly
> // created BB.
> - NumOfPGOSplit++;
> + IsCS ? NumOfCSPGOSplit++ : NumOfPGOSplit++;
> LLVM_DEBUG(dbgs() << "Split critical edge: " << getBBInfo(SrcBB).Index
> << " --> " << getBBInfo(DestBB).Index << "\n");
> unsigned SuccNum = GetSuccessorNumber(SrcBB, DestBB);
> @@ -720,12 +793,14 @@ BasicBlock *FuncPGOInstrumentation<Edge,
> // Critical edges will be split.
> static void instrumentOneFunc(
> Function &F, Module *M, BranchProbabilityInfo *BPI,
> BlockFrequencyInfo *BFI,
> - std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers) {
> + std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers,
> + bool IsCS) {
> // Split indirectbr critical edges here before computing the MST rather
> than
> // later in getInstrBB() to avoid invalidating it.
> SplitIndirectBrCriticalEdges(F, BPI, BFI);
> +
> FuncPGOInstrumentation<PGOEdge, BBInfo> FuncInfo(F, ComdatMembers,
> true, BPI,
> - BFI);
> + BFI, IsCS);
> unsigned NumCounters = FuncInfo.getNumCounters();
>
> uint32_t I = 0;
> @@ -852,10 +927,10 @@ public:
> PGOUseFunc(Function &Func, Module *Modu,
> std::unordered_multimap<Comdat *, GlobalValue *>
> &ComdatMembers,
> BranchProbabilityInfo *BPI = nullptr,
> - BlockFrequencyInfo *BFIin = nullptr)
> + BlockFrequencyInfo *BFIin = nullptr, bool IsCS = false)
> : F(Func), M(Modu), BFI(BFIin),
> - FuncInfo(Func, ComdatMembers, false, BPI, BFIin),
> - FreqAttr(FFA_Normal) {}
> + FuncInfo(Func, ComdatMembers, false, BPI, BFIin, IsCS),
> + FreqAttr(FFA_Normal), IsCS(IsCS) {}
>
> // Read counts for the instrumented BB from profile.
> bool readCounters(IndexedInstrProfReader *PGOReader, bool &AllZeros);
> @@ -928,6 +1003,9 @@ private:
> // Function hotness info derived from profile.
> FuncFreqAttr FreqAttr;
>
> + // Is to use the context sensitive profile.
> + bool IsCS;
> +
> // Find the Instrumented BB and set the value.
> void setInstrumentedCounts(const std::vector<uint64_t>
> &CountFromProfile);
>
> @@ -1021,23 +1099,31 @@ bool PGOUseFunc::readCounters(IndexedIns
> handleAllErrors(std::move(E), [&](const InstrProfError &IPE) {
> auto Err = IPE.get();
> bool SkipWarning = false;
> + LLVM_DEBUG(dbgs() << "Error in reading profile for Func "
> + << FuncInfo.FuncName << ": ");
> if (Err == instrprof_error::unknown_function) {
> - NumOfPGOMissing++;
> + IsCS ? NumOfCSPGOMissing++ : NumOfPGOMissing++;
> SkipWarning = !PGOWarnMissing;
> + LLVM_DEBUG(dbgs() << "unknown function");
> } else if (Err == instrprof_error::hash_mismatch ||
> Err == instrprof_error::malformed) {
> - NumOfPGOMismatch++;
> + IsCS ? NumOfCSPGOMismatch++ : NumOfPGOMismatch++;
> SkipWarning =
> NoPGOWarnMismatch ||
> (NoPGOWarnMismatchComdat &&
> (F.hasComdat() ||
> F.getLinkage() == GlobalValue::AvailableExternallyLinkage));
> + LLVM_DEBUG(dbgs() << "hash mismatch (skip=" << SkipWarning <<
> ")");
> }
>
> + LLVM_DEBUG(dbgs() << " IsCS=" << IsCS << "\n");
> if (SkipWarning)
> return;
>
> - std::string Msg = IPE.message() + std::string(" ") +
> F.getName().str();
> + std::string Msg = IPE.message() + std::string(" ") +
> F.getName().str() +
> + std::string(" Hash = ") +
> + std::to_string(FuncInfo.FunctionHash);
> +
> Ctx.diagnose(
> DiagnosticInfoPGOProfile(M->getName().data(), Msg, DS_Warning));
> });
> @@ -1046,7 +1132,7 @@ bool PGOUseFunc::readCounters(IndexedIns
> ProfileRecord = std::move(Result.get());
> std::vector<uint64_t> &CountFromProfile = ProfileRecord.Counts;
>
> - NumOfPGOFunc++;
> + IsCS ? NumOfCSPGOFunc++ : NumOfPGOFunc++;
> LLVM_DEBUG(dbgs() << CountFromProfile.size() << " counts\n");
> uint64_t ValueSum = 0;
> for (unsigned I = 0, S = CountFromProfile.size(); I < S; I++) {
> @@ -1061,7 +1147,11 @@ bool PGOUseFunc::readCounters(IndexedIns
> getBBInfo(nullptr).UnknownCountInEdge = 2;
>
> setInstrumentedCounts(CountFromProfile);
> +#if 0
> + ProgramMaxCount = PGOReader->getMaximumFunctionCount(IsCS);
> +#else
> ProgramMaxCount = PGOReader->getMaximumFunctionCount();
> +#endif
> return true;
> }
>
> @@ -1166,7 +1256,8 @@ void PGOUseFunc::populateCounters() {
> // Assign the scaled count values to the BB with multiple out edges.
> void PGOUseFunc::setBranchWeights() {
> // Generate MD_prof metadata for every branch instruction.
> - LLVM_DEBUG(dbgs() << "\nSetting branch weights.\n");
> + LLVM_DEBUG(dbgs() << "\nSetting branch weights for func " << F.getName()
> + << " IsCS=" << IsCS << "\n");
> for (auto &BB : F) {
> Instruction *TI = BB.getTerminator();
> if (TI->getNumSuccessors() < 2)
> @@ -1174,6 +1265,7 @@ void PGOUseFunc::setBranchWeights() {
> if (!(isa<BranchInst>(TI) || isa<SwitchInst>(TI) ||
> isa<IndirectBrInst>(TI)))
> continue;
> +
> if (getBBInfo(&BB).CountValue == 0)
> continue;
>
> @@ -1351,24 +1443,6 @@ void PGOUseFunc::annotateValueSites(uint
> }
> }
>
> -// Create a COMDAT variable INSTR_PROF_RAW_VERSION_VAR to make the runtime
> -// aware this is an ir_level profile so it can set the version flag.
> -static void createIRLevelProfileFlagVariable(Module &M) {
> - Type *IntTy64 = Type::getInt64Ty(M.getContext());
> - uint64_t ProfileVersion = (INSTR_PROF_RAW_VERSION |
> VARIANT_MASK_IR_PROF);
> - auto IRLevelVersionVariable = new GlobalVariable(
> - M, IntTy64, true, GlobalVariable::ExternalLinkage,
> - Constant::getIntegerValue(IntTy64, APInt(64, ProfileVersion)),
> - INSTR_PROF_QUOTE(INSTR_PROF_RAW_VERSION_VAR));
> - IRLevelVersionVariable->setVisibility(GlobalValue::DefaultVisibility);
> - Triple TT(M.getTargetTriple());
> - if (!TT.supportsCOMDAT())
> - IRLevelVersionVariable->setLinkage(GlobalValue::WeakAnyLinkage);
> - else
> - IRLevelVersionVariable->setComdat(M.getOrInsertComdat(
> - StringRef(INSTR_PROF_QUOTE(INSTR_PROF_RAW_VERSION_VAR))));
> -}
> -
> // Collect the set of members for each Comdat in module M and store
> // in ComdatMembers.
> static void collectComdatMembers(
> @@ -1389,8 +1463,11 @@ static void collectComdatMembers(
>
> static bool InstrumentAllFunctions(
> Module &M, function_ref<BranchProbabilityInfo *(Function &)>
> LookupBPI,
> - function_ref<BlockFrequencyInfo *(Function &)> LookupBFI) {
> - createIRLevelProfileFlagVariable(M);
> + function_ref<BlockFrequencyInfo *(Function &)> LookupBFI, bool IsCS) {
> + // For the context-sensitve instrumentation, we should have a separated
> pass
> + // (before LTO/ThinLTO linking) to create these variables.
> + if (!IsCS)
> + createIRLevelProfileFlagVar(M, /* IsCS */ false);
> std::unordered_multimap<Comdat *, GlobalValue *> ComdatMembers;
> collectComdatMembers(M, ComdatMembers);
>
> @@ -1399,7 +1476,7 @@ static bool InstrumentAllFunctions(
> continue;
> auto *BPI = LookupBPI(F);
> auto *BFI = LookupBFI(F);
> - instrumentOneFunc(F, &M, BPI, BFI, ComdatMembers);
> + instrumentOneFunc(F, &M, BPI, BFI, ComdatMembers, IsCS);
> }
> return true;
> }
> @@ -1414,7 +1491,7 @@ bool PGOInstrumentationGenLegacyPass::ru
> auto LookupBFI = [this](Function &F) {
> return &this->getAnalysis<BlockFrequencyInfoWrapperPass>(F).getBFI();
> };
> - return InstrumentAllFunctions(M, LookupBPI, LookupBFI);
> + return InstrumentAllFunctions(M, LookupBPI, LookupBFI, IsCS);
> }
>
> PreservedAnalyses PGOInstrumentationGen::run(Module &M,
> @@ -1428,7 +1505,7 @@ PreservedAnalyses PGOInstrumentationGen:
> return &FAM.getResult<BlockFrequencyAnalysis>(F);
> };
>
> - if (!InstrumentAllFunctions(M, LookupBPI, LookupBFI))
> + if (!InstrumentAllFunctions(M, LookupBPI, LookupBFI, IsCS))
> return PreservedAnalyses::all();
>
> return PreservedAnalyses::none();
> @@ -1437,7 +1514,7 @@ PreservedAnalyses PGOInstrumentationGen:
> static bool annotateAllFunctions(
> Module &M, StringRef ProfileFileName, StringRef
> ProfileRemappingFileName,
> function_ref<BranchProbabilityInfo *(Function &)> LookupBPI,
> - function_ref<BlockFrequencyInfo *(Function &)> LookupBFI) {
> + function_ref<BlockFrequencyInfo *(Function &)> LookupBFI, bool IsCS) {
> LLVM_DEBUG(dbgs() << "Read in profile counters: ");
> auto &Ctx = M.getContext();
> // Read the counter array from file.
> @@ -1458,6 +1535,11 @@ static bool annotateAllFunctions(
> StringRef("Cannot get
> PGOReader")));
> return false;
> }
> +#if 0
> + if (!PGOReader->hasCSIRLevelProfile() && IsCS)
> + return false;
> +#endif
> +
> // TODO: might need to change the warning once the clang option is
> finalized.
> if (!PGOReader->isIRLevelProfile()) {
> Ctx.diagnose(DiagnosticInfoPGOProfile(
> @@ -1477,7 +1559,7 @@ static bool annotateAllFunctions(
> // Split indirectbr critical edges here before computing the MST
> rather than
> // later in getInstrBB() to avoid invalidating it.
> SplitIndirectBrCriticalEdges(F, BPI, BFI);
> - PGOUseFunc Func(F, &M, ComdatMembers, BPI, BFI);
> + PGOUseFunc Func(F, &M, ComdatMembers, BPI, BFI, IsCS);
> bool AllZeros = false;
> if (!Func.readCounters(PGOReader.get(), AllZeros))
> continue;
> @@ -1525,7 +1607,14 @@ static bool annotateAllFunctions(
> }
> }
> }
> +#if 0
> + M.setProfileSummary(PGOReader->getSummary(IsCS).getMD(M.getContext()),
> + IsCS ? ProfileSummary::PSK_CSInstr
> + : ProfileSummary::PSK_Instr);
> +#else
> M.setProfileSummary(PGOReader->getSummary().getMD(M.getContext()));
> +#endif
> +
> // Set function hotness attribute from the profile.
> // We have to apply these attributes at the end because their presence
> // can affect the BranchProbabilityInfo of any callers, resulting in an
> @@ -1544,9 +1633,10 @@ static bool annotateAllFunctions(
> }
>
> PGOInstrumentationUse::PGOInstrumentationUse(std::string Filename,
> - std::string
> RemappingFilename)
> + std::string
> RemappingFilename,
> + bool IsCS)
> : ProfileFileName(std::move(Filename)),
> - ProfileRemappingFileName(std::move(RemappingFilename)) {
> + ProfileRemappingFileName(std::move(RemappingFilename)), IsCS(IsCS) {
> if (!PGOTestProfileFile.empty())
> ProfileFileName = PGOTestProfileFile;
> if (!PGOTestProfileRemappingFile.empty())
> @@ -1566,7 +1656,7 @@ PreservedAnalyses PGOInstrumentationUse:
> };
>
> if (!annotateAllFunctions(M, ProfileFileName, ProfileRemappingFileName,
> - LookupBPI, LookupBFI))
> + LookupBPI, LookupBFI, IsCS))
> return PreservedAnalyses::all();
>
> return PreservedAnalyses::none();
> @@ -1583,7 +1673,8 @@ bool PGOInstrumentationUseLegacyPass::ru
> return &this->getAnalysis<BlockFrequencyInfoWrapperPass>(F).getBFI();
> };
>
> - return annotateAllFunctions(M, ProfileFileName, "", LookupBPI,
> LookupBFI);
> + return annotateAllFunctions(M, ProfileFileName, "", LookupBPI,
> LookupBFI,
> + IsCS);
> }
>
> static std::string getSimpleNodeName(const BasicBlock *Node) {
>
> Modified: llvm/trunk/tools/gold/gold-plugin.cpp
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/tools/gold/gold-plugin.cpp?rev=354930&r1=354929&r2=354930&view=diff
>
> ==============================================================================
> --- llvm/trunk/tools/gold/gold-plugin.cpp (original)
> +++ llvm/trunk/tools/gold/gold-plugin.cpp Tue Feb 26 14:37:46 2019
> @@ -209,6 +209,10 @@ namespace options {
> static std::string OptRemarksFilename;
> static bool OptRemarksWithHotness = false;
>
> + // Context sensitive PGO options.
> + static std::string cs_profile_path;
> + static bool cs_pgo_gen = false;
> +
> static void process_plugin_option(const char *opt_)
> {
> if (opt_ == nullptr)
> @@ -268,7 +272,11 @@ namespace options {
> } else if (opt == "disable-verify") {
> DisableVerify = true;
> } else if (opt.startswith("sample-profile=")) {
> - sample_profile= opt.substr(strlen("sample-profile="));
> + sample_profile = opt.substr(strlen("sample-profile="));
> + } else if (opt == "cs-profile-generate") {
> + cs_pgo_gen = true;
> + } else if (opt.startswith("cs-profile-path=")) {
> + cs_profile_path = opt.substr(strlen("cs-profile-path="));
> } else if (opt == "new-pass-manager") {
> new_pass_manager = true;
> } else if (opt == "debug-pass-manager") {
> @@ -892,6 +900,10 @@ static std::unique_ptr<LTO> createLTO(In
> if (!options::sample_profile.empty())
> Conf.SampleProfile = options::sample_profile;
>
> + if (!options::cs_profile_path.empty())
> + Conf.CSIRProfile = options::cs_profile_path;
> + Conf.RunCSIRInstr = options::cs_pgo_gen;
> +
> Conf.DwoDir = options::dwo_dir;
>
> // Set up optimization remarks handling.
>
> Modified: llvm/trunk/tools/llvm-lto2/llvm-lto2.cpp
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/tools/llvm-lto2/llvm-lto2.cpp?rev=354930&r1=354929&r2=354930&view=diff
>
> ==============================================================================
> --- llvm/trunk/tools/llvm-lto2/llvm-lto2.cpp (original)
> +++ llvm/trunk/tools/llvm-lto2/llvm-lto2.cpp Tue Feb 26 14:37:46 2019
> @@ -104,6 +104,15 @@ static cl::opt<std::string>
> SamplePGOFile("lto-sample-profile-file",
> cl::desc("Specify a SamplePGO profile file"));
>
> +static cl::opt<std::string>
> + CSPGOFile("lto-cspgo-profile-file",
> + cl::desc("Specify a context sensitive PGO profile file"));
> +
> +static cl::opt<bool>
> + RunCSIRInstr("lto-cspgo-gen",
> + cl::desc("Run PGO context sensitive IR instrumentation"),
> + cl::init(false), cl::Hidden);
> +
> static cl::opt<bool>
> UseNewPM("use-new-pm",
> cl::desc("Run LTO passes using the new pass manager"),
> @@ -214,6 +223,8 @@ static int run(int argc, char **argv) {
> Conf.RemarksWithHotness = OptRemarksWithHotness;
>
> Conf.SampleProfile = SamplePGOFile;
> + Conf.CSIRProfile = CSPGOFile;
> + Conf.RunCSIRInstr = RunCSIRInstr;
>
> // Run a custom pipeline, if asked for.
> Conf.OptPipeline = OptPipeline;
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20190226/3cbab981/attachment-0001.html>
More information about the llvm-commits
mailing list