[flang-commits] [flang] [llvm] [CodeGen] Merge MachineFunctionSplitter into BasicBlockSections (PR #230365)
Rahman Lavaee via flang-commits
flang-commits at lists.llvm.org
Thu Oct 8 23:21:12 PDT 2026
https://github.com/rlavaee updated https://github.com/llvm/llvm-project/pull/230365
>From c2682c058123632f21702f1f243584844635923b Mon Sep 17 00:00:00 2001
From: Rahman Lavaee <rahmanl at google.com>
Date: Fri, 9 Oct 2026 01:50:10 +0000
Subject: [PATCH] [CodeGen] Merge MachineFunctionSplitter into
BasicBlockSections
Function splitting was done by two passes: MachineFunctionSplitter split
functions using their PGO/AutoFDO profile, and BasicBlockSections split
functions which have a basic block sections profile. MachineFunctionSplitter
had to skip functions in the basic block sections profile, and both passes
renumbered, sorted and fixed up the blocks of a function.
Move the PGO/AutoFDO splitting logic into BasicBlockSections, so that a
single pass does all function splitting. For every function,
BasicBlockSections now uses the basic block sections profile where it is
available, and otherwise, with -function-splitting=all, the PGO/AutoFDO
profile. The pass is added to the pipeline when function splitting is
enabled, and createBasicBlockSectionsPass() takes a SplitWithPGO
argument which makes it require MachineBlockFrequencyInfo and
ProfileSummaryInfo.
The splitting decisions and the resulting layout are unchanged. The
-mfs-psi-cutoff, -mfs-count-threshold and -mfs-split-ehcode options are
kept. createMachineFunctionSplitterPass() and the machine-function-splitter
pass name are removed.
---
.../Driver/split-machine-function-pass.f90 | 6 +-
llvm/docs/ReleaseNotes.md | 7 +
llvm/include/llvm/CodeGen/Passes.h | 11 +-
llvm/include/llvm/InitializePasses.h | 1 -
.../llvm/Passes/MachinePassRegistry.def | 1 -
llvm/include/llvm/Target/TargetOptions.h | 2 +-
llvm/lib/CodeGen/BasicBlockSections.cpp | 226 +++++++++++++++--
llvm/lib/CodeGen/CMakeLists.txt | 1 -
llvm/lib/CodeGen/CodeGen.cpp | 1 -
llvm/lib/CodeGen/MachineFunctionSplitter.cpp | 227 ------------------
llvm/lib/CodeGen/TargetPassConfig.cpp | 23 +-
llvm/test/CodeGen/X86/load-sample-profile.ll | 2 +-
.../gn/secondary/llvm/lib/CodeGen/BUILD.gn | 1 -
13 files changed, 232 insertions(+), 277 deletions(-)
delete mode 100644 llvm/lib/CodeGen/MachineFunctionSplitter.cpp
diff --git a/flang/test/Driver/split-machine-function-pass.f90 b/flang/test/Driver/split-machine-function-pass.f90
index b8659321c85d29..cfc192cf9db86b 100644
--- a/flang/test/Driver/split-machine-function-pass.f90
+++ b/flang/test/Driver/split-machine-function-pass.f90
@@ -1,4 +1,4 @@
-! Verify that the MachineFunctionSplitter pass is enabled while passing
+! Verify that the BasicBlockSections pass is enabled while passing
! -fsplit-machine-functions or -mllvm -function-splitting=all.
! REQUIRES: x86-registered-target
@@ -7,8 +7,8 @@
! RUN: %flang_fc1 -S %s -triple x86_64-unknown-linux-gnu -mllvm -function-splitting=all -mllvm -debug-pass=Structure -o /dev/null 2>&1 | FileCheck %s --check-prefix=SPLIT
! RUN: %flang_fc1 -S %s -triple x86_64-unknown-linux-gnu -mllvm -debug-pass=Structure -o /dev/null 2>&1 | FileCheck %s --check-prefix=NO-SPLIT
-! SPLIT: Machine Function Splitter Transformation
-! NO-SPLIT-NOT: Machine Function Splitter Transformation
+! SPLIT: Basic Block Sections Transformation
+! NO-SPLIT-NOT: Basic Block Sections Transformation
subroutine test
end subroutine test
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index 792622a3cc5895..be3b01e4c81d09 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -192,6 +192,13 @@ Makes programs 10x faster by doing Special New Thing.
replaced with `-fsplit-machine-functions` or
`-mllvm -function-splitting=all`.
+* Merged the `MachineFunctionSplitter` pass into the `BasicBlockSections`
+ pass, which now performs all function splitting: it uses the basic block
+ sections profile where available, and the PGO/AutoFDO profile elsewhere (with
+ `-function-splitting=all`). `createMachineFunctionSplitterPass()` and the
+ `machine-function-splitter` pass name are removed;
+ `createBasicBlockSectionsPass()` takes a `SplitWithPGO` argument instead.
+
### Changes to building LLVM
* A new `LLVM_ENABLE_LZMA` option (`ON`, `OFF` or `FORCE_ON`; default `ON`)
diff --git a/llvm/include/llvm/CodeGen/Passes.h b/llvm/include/llvm/CodeGen/Passes.h
index d5528d491ccefe..6623a60af7da03 100644
--- a/llvm/include/llvm/CodeGen/Passes.h
+++ b/llvm/include/llvm/CodeGen/Passes.h
@@ -62,8 +62,11 @@ LLVM_ABI FunctionPass *createUnreachableBlockEliminationPass();
LLVM_ABI MachineFunctionPass *createGCEmptyBasicBlocksLegacyPass();
/// createBasicBlockSections Pass - This pass assigns sections to machine
-/// basic blocks and is enabled with -fbasic-block-sections.
-LLVM_ABI MachineFunctionPass *createBasicBlockSectionsPass();
+/// basic blocks and is enabled with -fbasic-block-sections. If
+/// \p SplitWithPGO is true, it also splits functions without a basic block
+/// sections profile using their PGO/AutoFDO profile (-function-splitting=all).
+LLVM_ABI MachineFunctionPass *
+createBasicBlockSectionsPass(bool SplitWithPGO = false);
LLVM_ABI MachineFunctionPass *createBasicBlockPathCloningPass();
@@ -78,10 +81,6 @@ LLVM_ABI MachineFunctionPass *createInsertCodePrefetchPass();
/// createMachineBlockHashInfoPass - This pass computes basic block hashes.
LLVM_ABI MachineFunctionPass *createMachineBlockHashInfoPass();
-/// createMachineFunctionSplitterPass - This pass splits machine functions
-/// using profile information.
-LLVM_ABI MachineFunctionPass *createMachineFunctionSplitterPass();
-
/// createStaticDataSplitterPass - This is a machine-function pass that
/// categorizes static data hotness using profile information.
LLVM_ABI MachineFunctionPass *createStaticDataSplitterLegacyPass();
diff --git a/llvm/include/llvm/InitializePasses.h b/llvm/include/llvm/InitializePasses.h
index 458f3eee4f66d0..73820b2b02d5a1 100644
--- a/llvm/include/llvm/InitializePasses.h
+++ b/llvm/include/llvm/InitializePasses.h
@@ -201,7 +201,6 @@ LLVM_ABI void initializeMachineCycleInfoWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeMachineDominanceFrontierWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeMachineDominatorTreeWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeMachineFunctionPrinterPassPass(PassRegistry &);
-LLVM_ABI void initializeMachineFunctionSplitterPass(PassRegistry &);
LLVM_ABI void initializeMachineLateInstrsCleanupLegacyPass(PassRegistry &);
LLVM_ABI void initializeMachineLICMPass(PassRegistry &);
LLVM_ABI void initializeMachineLoopInfoWrapperPassPass(PassRegistry &);
diff --git a/llvm/include/llvm/Passes/MachinePassRegistry.def b/llvm/include/llvm/Passes/MachinePassRegistry.def
index ed582f2f61c650..fbb87e2c81ef13 100644
--- a/llvm/include/llvm/Passes/MachinePassRegistry.def
+++ b/llvm/include/llvm/Passes/MachinePassRegistry.def
@@ -256,7 +256,6 @@ DUMMY_MACHINE_FUNCTION_PASS("bbsections-profile-reader", BasicBlockSectionsProfi
DUMMY_MACHINE_FUNCTION_PASS("cfguard-longjmp", CFGuardLongjmpPass)
DUMMY_MACHINE_FUNCTION_PASS("fs-profile-loader", MIRProfileLoaderNewPass)
DUMMY_MACHINE_FUNCTION_PASS("lrshrink", LiveRangeShrinkPass)
-DUMMY_MACHINE_FUNCTION_PASS("machine-function-splitter", MachineFunctionSplitterPass)
DUMMY_MACHINE_FUNCTION_PASS("machineinstr-printer", MachineFunctionPrinterPass)
DUMMY_MACHINE_FUNCTION_PASS("mirfs-discriminators", MIRAddFSDiscriminatorsPass)
DUMMY_MACHINE_FUNCTION_PASS("prolog-epilog-code", PrologEpilogCodeInserterPass)
diff --git a/llvm/include/llvm/Target/TargetOptions.h b/llvm/include/llvm/Target/TargetOptions.h
index cd2d31fe2ae466..264db07ded991f 100644
--- a/llvm/include/llvm/Target/TargetOptions.h
+++ b/llvm/include/llvm/Target/TargetOptions.h
@@ -45,7 +45,7 @@ enum class BasicBlockSection {
// basic block sections for a subset of basic blocks which can be
// used to control object size bloats from creating sections.
Preset, // Similar to list but the blocks are identified by passes which
- // seek to use Basic Block Sections, e.g. MachineFunctionSplitter.
+ // seek to use Basic Block Sections, e.g. -function-splitting=all.
// This option cannot be set via the command line.
None // Do not use Basic Block Sections.
};
diff --git a/llvm/lib/CodeGen/BasicBlockSections.cpp b/llvm/lib/CodeGen/BasicBlockSections.cpp
index 761e827a93ea4e..d4102fe6140881 100644
--- a/llvm/lib/CodeGen/BasicBlockSections.cpp
+++ b/llvm/lib/CodeGen/BasicBlockSections.cpp
@@ -14,6 +14,11 @@
// and the rest are grouped in a cold section. The exception handling blocks are
// treated specially to ensure they are all in one seciton.
//
+// This pass also implements machine function splitting
+// (-function-splitting=all). Functions without a basic block sections profile
+// are split using PGO/AutoFDO profiles: cold blocks are moved into a separate
+// section with the symbol `foo.cold`.
+//
// Basic Block Sections
// ====================
//
@@ -66,20 +71,38 @@
// the corresponding basic blocks. This logic is implemented in AsmPrinter. This
// pass only assigns the BBSectionType of every function to ``labels``.
//
+// Machine Function Splitting
+// ==========================
+//
+// With -function-splitting=all, functions without a basic block sections
+// profile are split using PGO/AutoFDO profiles. Blocks deemed cold are placed
+// in a separate section prefixed with ".text.split." (see
+// -bbsections-cold-text-prefix), referenced by the symbol `foo.cold`. Grouping
+// cold blocks across functions decreases fragmentation and improves icache and
+// iTLB utilization. Such functions get the BBSectionType ``Preset``.
+//
+// For the original RFC of machine function splitting please see
+// https://groups.google.com/d/msg/llvm-dev/RUegaMg-iqc/wFAVxa6fCgAJ
+//
//===----------------------------------------------------------------------===//
#include "llvm/ADT/SmallVector.h"
#include "llvm/ADT/StringRef.h"
+#include "llvm/Analysis/EHUtils.h"
+#include "llvm/Analysis/ProfileSummaryInfo.h"
#include "llvm/CodeGen/BasicBlockMatchingAndInference.h"
#include "llvm/CodeGen/BasicBlockSectionUtils.h"
#include "llvm/CodeGen/BasicBlockSectionsProfileReader.h"
+#include "llvm/CodeGen/MachineBlockFrequencyInfo.h"
#include "llvm/CodeGen/MachineDominators.h"
#include "llvm/CodeGen/MachineFunction.h"
#include "llvm/CodeGen/MachineFunctionPass.h"
#include "llvm/CodeGen/MachinePostDominators.h"
#include "llvm/CodeGen/Passes.h"
#include "llvm/CodeGen/TargetInstrInfo.h"
+#include "llvm/IR/Function.h"
#include "llvm/InitializePasses.h"
+#include "llvm/Support/CommandLine.h"
#include "llvm/Support/UniqueBBID.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/Utils/CodeLayout.h"
@@ -102,18 +125,43 @@ static cl::opt<bool> BBSectionsDetectSourceDrift(
"mismatch for this function"),
cl::init(true), cl::Hidden);
+// FIXME: This cutoff value is CPU dependent and should be moved to
+// TargetTransformInfo once we consider enabling this on other platforms.
+// The value is expressed as a ProfileSummaryInfo integer percentile cutoff.
+// Defaults to 999950, i.e. all blocks colder than 99.995 percentile are split.
+// The default was empirically determined to be optimal when considering cutoff
+// values between 99%-ile to 100%-ile with respect to iTLB and icache metrics on
+// Intel CPUs.
+static cl::opt<unsigned>
+ PercentileCutoff("mfs-psi-cutoff",
+ cl::desc("Percentile profile summary cutoff used to "
+ "determine cold blocks. Unused if set to zero."),
+ cl::init(999950), cl::Hidden);
+
+static cl::opt<unsigned> ColdCountThreshold(
+ "mfs-count-threshold",
+ cl::desc(
+ "Minimum number of times a block must be executed to be retained."),
+ cl::init(1), cl::Hidden);
+
+static cl::opt<bool> SplitAllEHCode(
+ "mfs-split-ehcode",
+ cl::desc("Splits all EH code and it's descendants by default."),
+ cl::init(false), cl::Hidden);
+
namespace {
class BasicBlockSections : public MachineFunctionPass {
public:
static char ID;
- BasicBlockSections() : MachineFunctionPass(ID) {
+ explicit BasicBlockSections(bool SplitWithPGO = false)
+ : MachineFunctionPass(ID), SplitWithPGO(SplitWithPGO) {
initializeBasicBlockSectionsPass(*PassRegistry::getPassRegistry());
}
StringRef getPassName() const override {
- return "Basic Block Sections Analysis";
+ return "Basic Block Sections Transformation";
}
void getAnalysisUsage(AnalysisUsage &AU) const override;
@@ -125,6 +173,12 @@ class BasicBlockSections : public MachineFunctionPass {
private:
bool handleBBSections(MachineFunction &MF);
bool handleBBAddrMap(MachineFunction &MF);
+ bool shouldSplitWithPGO(MachineFunction &MF);
+ void assignColdSectionWithPGO(MachineFunction &MF);
+
+ // Whether functions without a basic block sections profile are split using
+ // PGO/AutoFDO profiles (-function-splitting=all).
+ const bool SplitWithPGO;
};
} // end anonymous namespace
@@ -136,6 +190,8 @@ INITIALIZE_PASS_BEGIN(
"into clusters of basic blocks.",
false, false)
INITIALIZE_PASS_DEPENDENCY(BasicBlockSectionsProfileReaderWrapperPass)
+INITIALIZE_PASS_DEPENDENCY(MachineBlockFrequencyInfoWrapperPass)
+INITIALIZE_PASS_DEPENDENCY(ProfileSummaryInfoWrapperPass)
INITIALIZE_PASS_END(BasicBlockSections, "bbsections-prepare",
"Prepares for basic block sections, by splitting functions "
"into clusters of basic blocks.",
@@ -375,13 +431,128 @@ bool llvm::hasInstrProfHashMismatch(MachineFunction &MF) {
return false;
}
+/// setDescendantEHBlocksCold - This splits all EH pads and blocks reachable
+/// only by EH pad as cold. This will help mark EH pads statically cold
+/// instead of relying on profile data.
+static void setDescendantEHBlocksCold(MachineFunction &MF) {
+ DenseSet<MachineBasicBlock *> EHBlocks;
+ computeEHOnlyBlocks(MF, EHBlocks);
+ for (auto Block : EHBlocks) {
+ Block->setSectionID(MBBSectionID::ColdSectionID);
+ }
+}
+
+static bool isColdBlock(const MachineBasicBlock &MBB,
+ const MachineBlockFrequencyInfo *MBFI,
+ ProfileSummaryInfo *PSI) {
+ std::optional<uint64_t> Count = MBFI->getBlockProfileCount(&MBB);
+ // For instrumentation profiles and sample profiles, we use different ways
+ // to judge whether a block is cold and should be split.
+ if (PSI->hasInstrumentationProfile() || PSI->hasCSInstrumentationProfile()) {
+ // If using instrument profile, which is deemed "accurate", no count means
+ // cold.
+ if (!Count)
+ return true;
+ if (PercentileCutoff > 0)
+ return PSI->isColdCountNthPercentile(PercentileCutoff, *Count);
+ // Fallthrough to end of function.
+ } else if (PSI->hasSampleProfile()) {
+ // For sample profile, no count means "do not judege coldness".
+ if (!Count)
+ return false;
+ }
+
+ return (*Count < ColdCountThreshold);
+}
+
+// Returns whether a function which does not have a basic block sections profile
+// should be split using its PGO/AutoFDO profile.
+bool BasicBlockSections::shouldSplitWithPGO(MachineFunction &MF) {
+ if (!SplitWithPGO || skipFunction(MF.getFunction()))
+ return false;
+
+ // We target functions with profile data. Static information in the form
+ // of exception handling code may be split to cold if user passes the
+ // mfs-split-ehcode flag.
+ if (!MF.getFunction().hasProfileData() && !SplitAllEHCode)
+ return false;
+
+ const TargetInstrInfo &TII = *MF.getSubtarget().getInstrInfo();
+ if (!TII.isFunctionSafeToSplit(MF))
+ return false;
+
+ // Do not split functions which are in the basic block sections profile, even
+ // if their profile is not used due to source drift.
+ if (MF.getTarget().getBBSectionsType() == BasicBlockSection::List &&
+ getAnalysis<BasicBlockSectionsProfileReaderWrapperPass>()
+ .getBBSPR()
+ .isFunctionHot(MF.getName()))
+ return false;
+
+ return true;
+}
+
+// Splits a function using its PGO/AutoFDO profile: assigns the cold section to
+// cold blocks, and the default section to all the other blocks.
+void BasicBlockSections::assignColdSectionWithPGO(MachineFunction &MF) {
+ MF.setBBSectionsType(BasicBlockSection::Preset);
+
+ const TargetInstrInfo &TII = *MF.getSubtarget().getInstrInfo();
+ bool UseProfileData = MF.getFunction().hasProfileData();
+ MachineBlockFrequencyInfo *MBFI = nullptr;
+ ProfileSummaryInfo *PSI = nullptr;
+ if (UseProfileData) {
+ MBFI = &getAnalysis<MachineBlockFrequencyInfoWrapperPass>().getMBFI();
+ PSI = &getAnalysis<ProfileSummaryInfoWrapperPass>().getPSI();
+ // If we don't have a good profile (sample profile is not deemed
+ // as a "good profile") and the function is not hot, then early
+ // return. (Because we can only trust hot functions when profile
+ // quality is not good.)
+ if (PSI->hasSampleProfile() && !PSI->isFunctionHotInCallGraph(&MF, *MBFI)) {
+ // Split all EH code and it's descendant statically by default.
+ if (SplitAllEHCode)
+ setDescendantEHBlocksCold(MF);
+ return;
+ }
+ }
+
+ SmallVector<MachineBasicBlock *, 2> LandingPads;
+ for (auto &MBB : MF) {
+ if (MBB.isEntryBlock())
+ continue;
+
+ if (MBB.isEHPad())
+ LandingPads.push_back(&MBB);
+ else if (UseProfileData && isColdBlock(MBB, MBFI, PSI) &&
+ TII.isMBBSafeToSplitToCold(MBB) && !SplitAllEHCode)
+ MBB.setSectionID(MBBSectionID::ColdSectionID);
+ }
+
+ // Split all EH code and it's descendant statically by default.
+ if (SplitAllEHCode)
+ setDescendantEHBlocksCold(MF);
+ // We only split out eh pads if all of them are cold.
+ else {
+ // Here we have UseProfileData == true.
+ bool HasHotLandingPads = false;
+ for (const MachineBasicBlock *LP : LandingPads) {
+ if (!isColdBlock(*LP, MBFI, PSI) || !TII.isMBBSafeToSplitToCold(*LP))
+ HasHotLandingPads = true;
+ }
+ if (!HasHotLandingPads) {
+ for (MachineBasicBlock *LP : LandingPads)
+ LP->setSectionID(MBBSectionID::ColdSectionID);
+ }
+ }
+}
+
// Identify, arrange, and modify basic blocks which need separate sections
-// according to the specification provided by the -fbasic-block-sections flag.
+// according to the specification provided by the -fbasic-block-sections flag,
+// or according to the PGO/AutoFDO profile with -function-splitting=all.
bool BasicBlockSections::handleBBSections(MachineFunction &MF) {
auto BBSectionsType = MF.getTarget().getBBSectionsType();
- if (BBSectionsType == BasicBlockSection::None)
- return false;
+ DenseMap<UniqueBBID, BBClusterInfo> FuncClusterInfo;
// Check for source drift. If the source has changed since the profiles
// were obtained, optimizing basic blocks might be sub-optimal.
// This only applies to BasicBlockSection::List as it creates
@@ -389,11 +560,7 @@ bool BasicBlockSections::handleBBSections(MachineFunction &MF) {
// invalidate these groupings leading to sub-optimal code generation with
// regards to performance.
if (BBSectionsType == BasicBlockSection::List &&
- hasInstrProfHashMismatch(MF))
- return false;
-
- DenseMap<UniqueBBID, BBClusterInfo> FuncClusterInfo;
- if (BBSectionsType == BasicBlockSection::List) {
+ !hasInstrProfHashMismatch(MF)) {
SmallVector<BBClusterInfo> ClusterInfo;
if (auto *BMI = getAnalysisIfAvailable<BasicBlockMatchingAndInference>()) {
ClusterInfo = createBBClusterInfoForFunction(MF, *BMI);
@@ -401,19 +568,32 @@ bool BasicBlockSections::handleBBSections(MachineFunction &MF) {
ClusterInfo = getAnalysis<BasicBlockSectionsProfileReaderWrapperPass>()
.getClusterInfoForFunction(MF.getName());
}
- if (ClusterInfo.empty())
- return false;
for (auto &BBClusterInfo : ClusterInfo) {
FuncClusterInfo.try_emplace(BBClusterInfo.BBID, BBClusterInfo);
}
}
- // Renumber blocks before sorting them. This is useful for accessing the
- // original layout positions and finding the original fallthroughs.
+ // A function without a basic block sections profile is split using its
+ // PGO/AutoFDO profile, if enabled.
+ const bool UsePGO =
+ BBSectionsType == BasicBlockSection::None ||
+ (BBSectionsType == BasicBlockSection::List && FuncClusterInfo.empty());
+ if (UsePGO && !shouldSplitWithPGO(MF))
+ return false;
+
+ // Renumber blocks before assigning sections and sorting them. This is useful
+ // for accessing the original layout positions and finding the original
+ // fallthroughs. Blocks which are not in any cluster keep their original
+ // order, which retains decisions made by prior passes such as
+ // MachineBlockPlacement.
MF.RenumberBlocks();
- MF.setBBSectionsType(BBSectionsType);
- assignSections(MF, FuncClusterInfo);
+ if (UsePGO) {
+ assignColdSectionWithPGO(MF);
+ } else {
+ MF.setBBSectionsType(BBSectionsType);
+ assignSections(MF, FuncClusterInfo);
+ }
const MachineBasicBlock &EntryBB = MF.front();
auto EntryBBSectionID = EntryBB.getSectionID();
@@ -452,8 +632,10 @@ bool BasicBlockSections::handleBBSections(MachineFunction &MF) {
// their cluster and their position within the cluster. A section holds more
// than one cluster only with -function-splitting=none. Basic blocks which
// are not in any cluster come after all the profiled basic blocks of the
- // section, in their original order.
- if (XSectionID.Type == MBBSectionID::SectionType::Default) {
+ // section, in their original order. Without a basic block sections profile,
+ // all basic blocks keep their original order (they might not have BBIDs).
+ if (XSectionID.Type == MBBSectionID::SectionType::Default &&
+ !FuncClusterInfo.empty()) {
auto XI = FuncClusterInfo.find(*X.getBBID());
auto YI = FuncClusterInfo.find(*Y.getBBID());
bool XInCluster = XI != FuncClusterInfo.end();
@@ -494,12 +676,16 @@ bool BasicBlockSections::runOnMachineFunction(MachineFunction &MF) {
void BasicBlockSections::getAnalysisUsage(AnalysisUsage &AU) const {
AU.setPreservesAll();
AU.addRequired<BasicBlockSectionsProfileReaderWrapperPass>();
+ if (SplitWithPGO) {
+ AU.addRequired<MachineBlockFrequencyInfoWrapperPass>();
+ AU.addRequired<ProfileSummaryInfoWrapperPass>();
+ }
AU.addUsedIfAvailable<BasicBlockMatchingAndInference>();
AU.addUsedIfAvailable<MachineDominatorTreeWrapperPass>();
AU.addUsedIfAvailable<MachinePostDominatorTreeWrapperPass>();
MachineFunctionPass::getAnalysisUsage(AU);
}
-MachineFunctionPass *llvm::createBasicBlockSectionsPass() {
- return new BasicBlockSections();
+MachineFunctionPass *llvm::createBasicBlockSectionsPass(bool SplitWithPGO) {
+ return new BasicBlockSections(SplitWithPGO);
}
diff --git a/llvm/lib/CodeGen/CMakeLists.txt b/llvm/lib/CodeGen/CMakeLists.txt
index 211ce644a4162d..b2cdd59ba4edfe 100644
--- a/llvm/lib/CodeGen/CMakeLists.txt
+++ b/llvm/lib/CodeGen/CMakeLists.txt
@@ -144,7 +144,6 @@ add_llvm_component_library(LLVMCodeGen
MachineFunction.cpp
MachineFunctionPass.cpp
MachineFunctionPrinterPass.cpp
- MachineFunctionSplitter.cpp
MachineIDFSSAUpdater.cpp
MachineInstrBundle.cpp
MachineInstr.cpp
diff --git a/llvm/lib/CodeGen/CodeGen.cpp b/llvm/lib/CodeGen/CodeGen.cpp
index bdf87c3e9ef404..0a2ec7d45cf768 100644
--- a/llvm/lib/CodeGen/CodeGen.cpp
+++ b/llvm/lib/CodeGen/CodeGen.cpp
@@ -95,7 +95,6 @@ void llvm::initializeCodeGen(PassRegistry &Registry) {
initializeMachineCycleInfoWrapperPassPass(Registry);
initializeMachineDominatorTreeWrapperPassPass(Registry);
initializeMachineFunctionPrinterPassPass(Registry);
- initializeMachineFunctionSplitterPass(Registry);
initializeMachineLateInstrsCleanupLegacyPass(Registry);
initializeMachineLICMPass(Registry);
initializeMachineLoopInfoWrapperPassPass(Registry);
diff --git a/llvm/lib/CodeGen/MachineFunctionSplitter.cpp b/llvm/lib/CodeGen/MachineFunctionSplitter.cpp
deleted file mode 100644
index 6adda8a345cf40..00000000000000
--- a/llvm/lib/CodeGen/MachineFunctionSplitter.cpp
+++ /dev/null
@@ -1,227 +0,0 @@
-//===-- MachineFunctionSplitter.cpp - Split machine functions //-----------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// \file
-// Uses profile information to split out cold blocks.
-//
-// This pass splits out cold machine basic blocks from the parent function. This
-// implementation leverages the basic block section framework. Blocks marked
-// cold by this pass are grouped together in a separate section prefixed with
-// ".text.unlikely.*". The linker can then group these together as a cold
-// section. The split part of the function is a contiguous region identified by
-// the symbol "foo.cold". Grouping all cold blocks across functions together
-// decreases fragmentation and improves icache and itlb utilization. Note that
-// the overall changes to the binary size are negligible; only a small number of
-// additional jump instructions may be introduced.
-//
-// For the original RFC of this pass please see
-// https://groups.google.com/d/msg/llvm-dev/RUegaMg-iqc/wFAVxa6fCgAJ
-//===----------------------------------------------------------------------===//
-
-#include "llvm/ADT/SmallVector.h"
-#include "llvm/Analysis/BranchProbabilityInfo.h"
-#include "llvm/Analysis/EHUtils.h"
-#include "llvm/Analysis/ProfileSummaryInfo.h"
-#include "llvm/CodeGen/BasicBlockSectionUtils.h"
-#include "llvm/CodeGen/BasicBlockSectionsProfileReader.h"
-#include "llvm/CodeGen/MachineBasicBlock.h"
-#include "llvm/CodeGen/MachineBlockFrequencyInfo.h"
-#include "llvm/CodeGen/MachineFunction.h"
-#include "llvm/CodeGen/MachineFunctionPass.h"
-#include "llvm/CodeGen/MachineModuleInfo.h"
-#include "llvm/CodeGen/Passes.h"
-#include "llvm/CodeGen/TargetInstrInfo.h"
-#include "llvm/IR/Function.h"
-#include "llvm/InitializePasses.h"
-#include "llvm/Support/CommandLine.h"
-#include <optional>
-
-using namespace llvm;
-
-// FIXME: This cutoff value is CPU dependent and should be moved to
-// TargetTransformInfo once we consider enabling this on other platforms.
-// The value is expressed as a ProfileSummaryInfo integer percentile cutoff.
-// Defaults to 999950, i.e. all blocks colder than 99.995 percentile are split.
-// The default was empirically determined to be optimal when considering cutoff
-// values between 99%-ile to 100%-ile with respect to iTLB and icache metrics on
-// Intel CPUs.
-static cl::opt<unsigned>
- PercentileCutoff("mfs-psi-cutoff",
- cl::desc("Percentile profile summary cutoff used to "
- "determine cold blocks. Unused if set to zero."),
- cl::init(999950), cl::Hidden);
-
-static cl::opt<unsigned> ColdCountThreshold(
- "mfs-count-threshold",
- cl::desc(
- "Minimum number of times a block must be executed to be retained."),
- cl::init(1), cl::Hidden);
-
-static cl::opt<bool> SplitAllEHCode(
- "mfs-split-ehcode",
- cl::desc("Splits all EH code and it's descendants by default."),
- cl::init(false), cl::Hidden);
-
-namespace {
-
-class MachineFunctionSplitter : public MachineFunctionPass {
-public:
- static char ID;
- MachineFunctionSplitter() : MachineFunctionPass(ID) {}
-
- StringRef getPassName() const override {
- return "Machine Function Splitter Transformation";
- }
-
- void getAnalysisUsage(AnalysisUsage &AU) const override;
-
- bool runOnMachineFunction(MachineFunction &F) override;
-};
-} // end anonymous namespace
-
-/// setDescendantEHBlocksCold - This splits all EH pads and blocks reachable
-/// only by EH pad as cold. This will help mark EH pads statically cold
-/// instead of relying on profile data.
-static void setDescendantEHBlocksCold(MachineFunction &MF) {
- DenseSet<MachineBasicBlock *> EHBlocks;
- computeEHOnlyBlocks(MF, EHBlocks);
- for (auto Block : EHBlocks) {
- Block->setSectionID(MBBSectionID::ColdSectionID);
- }
-}
-
-static void finishAdjustingBasicBlocksAndLandingPads(MachineFunction &MF) {
- auto Comparator = [](const MachineBasicBlock &X, const MachineBasicBlock &Y) {
- return X.getSectionID().Type < Y.getSectionID().Type;
- };
- llvm::sortBasicBlocksAndUpdateBranches(MF, Comparator);
- llvm::avoidZeroOffsetLandingPad(MF);
-}
-
-static bool isColdBlock(const MachineBasicBlock &MBB,
- const MachineBlockFrequencyInfo *MBFI,
- ProfileSummaryInfo *PSI) {
- std::optional<uint64_t> Count = MBFI->getBlockProfileCount(&MBB);
- // For instrumentation profiles and sample profiles, we use different ways
- // to judge whether a block is cold and should be split.
- if (PSI->hasInstrumentationProfile() || PSI->hasCSInstrumentationProfile()) {
- // If using instrument profile, which is deemed "accurate", no count means
- // cold.
- if (!Count)
- return true;
- if (PercentileCutoff > 0)
- return PSI->isColdCountNthPercentile(PercentileCutoff, *Count);
- // Fallthrough to end of function.
- } else if (PSI->hasSampleProfile()) {
- // For sample profile, no count means "do not judege coldness".
- if (!Count)
- return false;
- }
-
- return (*Count < ColdCountThreshold);
-}
-
-bool MachineFunctionSplitter::runOnMachineFunction(MachineFunction &MF) {
- if (skipFunction(MF.getFunction()))
- return false;
-
- // Do not split functions when -basic-block-sections=all is specified.
- if (MF.getTarget().getBBSectionsType() == llvm::BasicBlockSection::All)
- return false;
- // We target functions with profile data. Static information in the form
- // of exception handling code may be split to cold if user passes the
- // mfs-split-ehcode flag.
- bool UseProfileData = MF.getFunction().hasProfileData();
- if (!UseProfileData && !SplitAllEHCode)
- return false;
-
- const TargetInstrInfo &TII = *MF.getSubtarget().getInstrInfo();
- if (!TII.isFunctionSafeToSplit(MF))
- return false;
-
- // Do not split functions with BasicBlockSections profiles as they will
- // be split by the BasicBlockSections pass.
- auto BBSectionsProfile =
- getAnalysisIfAvailable<BasicBlockSectionsProfileReaderWrapperPass>();
- if (BBSectionsProfile != nullptr &&
- BBSectionsProfile->getBBSPR().isFunctionHot(MF.getName()))
- return false;
-
- // Renumbering blocks here preserves the order of the blocks as
- // sortBasicBlocksAndUpdateBranches uses the numeric identifier to sort
- // blocks. Preserving the order of blocks is essential to retaining decisions
- // made by prior passes such as MachineBlockPlacement.
- MF.RenumberBlocks();
- MF.setBBSectionsType(BasicBlockSection::Preset);
-
- MachineBlockFrequencyInfo *MBFI = nullptr;
- ProfileSummaryInfo *PSI = nullptr;
- if (UseProfileData) {
- MBFI = &getAnalysis<MachineBlockFrequencyInfoWrapperPass>().getMBFI();
- PSI = &getAnalysis<ProfileSummaryInfoWrapperPass>().getPSI();
- // If we don't have a good profile (sample profile is not deemed
- // as a "good profile") and the function is not hot, then early
- // return. (Because we can only trust hot functions when profile
- // quality is not good.)
- if (PSI->hasSampleProfile() && !PSI->isFunctionHotInCallGraph(&MF, *MBFI)) {
- // Split all EH code and it's descendant statically by default.
- if (SplitAllEHCode)
- setDescendantEHBlocksCold(MF);
- finishAdjustingBasicBlocksAndLandingPads(MF);
- return true;
- }
- }
-
- SmallVector<MachineBasicBlock *, 2> LandingPads;
- for (auto &MBB : MF) {
- if (MBB.isEntryBlock())
- continue;
-
- if (MBB.isEHPad())
- LandingPads.push_back(&MBB);
- else if (UseProfileData && isColdBlock(MBB, MBFI, PSI) &&
- TII.isMBBSafeToSplitToCold(MBB) && !SplitAllEHCode)
- MBB.setSectionID(MBBSectionID::ColdSectionID);
- }
-
- // Split all EH code and it's descendant statically by default.
- if (SplitAllEHCode)
- setDescendantEHBlocksCold(MF);
- // We only split out eh pads if all of them are cold.
- else {
- // Here we have UseProfileData == true.
- bool HasHotLandingPads = false;
- for (const MachineBasicBlock *LP : LandingPads) {
- if (!isColdBlock(*LP, MBFI, PSI) || !TII.isMBBSafeToSplitToCold(*LP))
- HasHotLandingPads = true;
- }
- if (!HasHotLandingPads) {
- for (MachineBasicBlock *LP : LandingPads)
- LP->setSectionID(MBBSectionID::ColdSectionID);
- }
- }
-
- finishAdjustingBasicBlocksAndLandingPads(MF);
- return true;
-}
-
-void MachineFunctionSplitter::getAnalysisUsage(AnalysisUsage &AU) const {
- AU.addRequired<MachineModuleInfoWrapperPass>();
- AU.addRequired<MachineBlockFrequencyInfoWrapperPass>();
- AU.addRequired<ProfileSummaryInfoWrapperPass>();
- AU.addUsedIfAvailable<BasicBlockSectionsProfileReaderWrapperPass>();
-}
-
-char MachineFunctionSplitter::ID = 0;
-INITIALIZE_PASS(MachineFunctionSplitter, "machine-function-splitter",
- "Split machine functions using profile information", false,
- false)
-
-MachineFunctionPass *llvm::createMachineFunctionSplitterPass() {
- return new MachineFunctionSplitter();
-}
diff --git a/llvm/lib/CodeGen/TargetPassConfig.cpp b/llvm/lib/CodeGen/TargetPassConfig.cpp
index 44846365580eed..33456dc4fe9686 100644
--- a/llvm/lib/CodeGen/TargetPassConfig.cpp
+++ b/llvm/lib/CodeGen/TargetPassConfig.cpp
@@ -1092,10 +1092,10 @@ void TargetPassConfig::addMachinePasses() {
addPass(createMIRAddFSDiscriminatorsPass(
sampleprof::FSDiscriminatorPass::PassLast));
- const bool SplitFunctions =
+ const bool SplitFunctionsWithPGO =
TM->Options.FunctionSplitting == FunctionSplittingMode::All;
- if (SplitFunctions || Opts.split_static_data ||
+ if (SplitFunctionsWithPGO || Opts.split_static_data ||
TM->Options.EnableStaticDataPartitioning) {
const std::string ProfileFile = getFSProfileFile(TM);
if (!ProfileFile.empty()) {
@@ -1113,14 +1113,6 @@ void TargetPassConfig::addMachinePasses() {
}
}
- // Machine function splitter uses the basic block sections feature.
- // When used along with `-basic-block-sections=`, the basic-block-sections
- // feature takes precedence. This means functions eligible for
- // basic-block-sections optimizations (`=all`, or `=list=` with function
- // included in the list profile) will get that optimization instead.
- if (SplitFunctions)
- addPass(createMachineFunctionSplitterPass());
-
if (Opts.split_static_data || TM->Options.EnableStaticDataPartitioning) {
// The static data splitter pass is a machine function pass. and
// static data annotator pass is a module-wide pass. See the file comment
@@ -1128,10 +1120,13 @@ void TargetPassConfig::addMachinePasses() {
addPass(createStaticDataSplitterLegacyPass());
addPass(createStaticDataAnnotatorLegacyPass());
}
- // We run the BasicBlockSections pass if either we need BB sections or BB
- // address map (or both).
+ // We run the BasicBlockSections pass if we need BB sections, BB address map,
+ // or function splitting. With function splitting, functions which are
+ // eligible for basic-block-sections optimizations (`=all`, or `=list=` with
+ // function included in the list profile) get that optimization, and the other
+ // functions are split using their PGO/AutoFDO profile.
if (TM->getBBSectionsType() != llvm::BasicBlockSection::None ||
- TM->Options.BBAddrMap) {
+ TM->Options.BBAddrMap || SplitFunctionsWithPGO) {
if (shouldEmitBBHash() || Opts.basic_block_section_match_infer)
addPass(llvm::createMachineBlockHashInfoPass());
if (TM->getBBSectionsType() == llvm::BasicBlockSection::List) {
@@ -1144,7 +1139,7 @@ void TargetPassConfig::addMachinePasses() {
addPass(llvm::createInsertCodePrefetchPass());
}
}
- addPass(llvm::createBasicBlockSectionsPass());
+ addPass(llvm::createBasicBlockSectionsPass(SplitFunctionsWithPGO));
}
addPostBBSections();
diff --git a/llvm/test/CodeGen/X86/load-sample-profile.ll b/llvm/test/CodeGen/X86/load-sample-profile.ll
index 3ac1f39b17e388..e65aa98123b3bf 100644
--- a/llvm/test/CodeGen/X86/load-sample-profile.ll
+++ b/llvm/test/CodeGen/X86/load-sample-profile.ll
@@ -26,7 +26,7 @@
; PROFILE-MFS: SampleFDO loader in MIR
; PROFILE-MFS: Add FS discriminators in MIR
; PROFILE-MFS: SampleFDO loader in MIR
-; PROFILE-MFS: Machine Function Splitter Transformation
+; PROFILE-MFS: Basic Block Sections Transformation
define void @foo4(i1 zeroext %0, i1 zeroext %1) nounwind {
br i1 %0, label %3, label %7
diff --git a/llvm/utils/gn/secondary/llvm/lib/CodeGen/BUILD.gn b/llvm/utils/gn/secondary/llvm/lib/CodeGen/BUILD.gn
index dfcb12d75a4657..86fcc2e5a3bd7c 100644
--- a/llvm/utils/gn/secondary/llvm/lib/CodeGen/BUILD.gn
+++ b/llvm/utils/gn/secondary/llvm/lib/CodeGen/BUILD.gn
@@ -152,7 +152,6 @@ static_library("CodeGen") {
"MachineFunctionAnalysis.cpp",
"MachineFunctionPass.cpp",
"MachineFunctionPrinterPass.cpp",
- "MachineFunctionSplitter.cpp",
"MachineIDFSSAUpdater.cpp",
"MachineInstr.cpp",
"MachineInstrBundle.cpp",
More information about the flang-commits
mailing list