[llvm] [CodeGen][StaticDataPartitioning]Place local-linkage global variables in hot or unlikely prefixed sections based on profile information (PR #125756)

[Mingming Liu via llvm-commits]

================
@@ -117,18 +142,89 @@ bool StaticDataSplitter::partitionStaticDataWithProfiles(MachineFunction &MF) {
           // Hotness is based on source basic block hotness.
           // TODO: PSI APIs are about instruction hotness. Introduce API for
           // data access hotness.
-          if (PSI->isColdBlock(&MBB, MBFI))
+          if (Count && PSI->isColdCount(*Count))
             Hotness = MachineFunctionDataHotness::Cold;
 
           if (MJTI->updateJumpTableEntryHotness(JTI, Hotness))
             ++NumChangedJumpTables;
+        } else if (Op.isGlobal()) {
+          // Find global variables with local linkage
+          const GlobalVariable *GV =
+              getLocalLinkageGlobalVariable(Op.getGlobal());
+          if (!GV || !inStaticDataSection(GV, TM))
+            continue;
+
+          // Acccumulate data profile count across machine function
+          // instructions.
+          // TODO: Analyze global variable's initializers.
+          if (Count) {
+            auto [It, Inserted] =
+                DataProfileCounts.try_emplace(GV, APInt(128, 0));
+            It->second += *Count;
+          }
         }
       }
     }
   }
   return NumChangedJumpTables > 0;
 }
 
+const GlobalVariable *
+StaticDataSplitter::getLocalLinkageGlobalVariable(const GlobalValue *GV) {
+  if (!GV || GV->isDeclarationForLinker())
+    return nullptr;
+
+  return GV->hasLocalLinkage() ? dyn_cast<GlobalVariable>(GV) : nullptr;
+}
+
+bool StaticDataSplitter::inStaticDataSection(const GlobalVariable *GV,
+                                             const TargetMachine &TM) {
+  assert(GV && "Caller guaranteed");
+
+  // Skip LLVM reserved symbols.
+  if (GV->getName().starts_with("llvm."))
+    return false;
+
+  SectionKind Kind = TargetLoweringObjectFile::getKindForGlobal(GV, TM);
+  return Kind.isData() || Kind.isReadOnly() || Kind.isReadOnlyWithRel() ||
+         Kind.isBSS();
+}
+
+bool StaticDataSplitter::updateGlobalVariableSectionPrefix(
+    MachineFunction &MF) {
+  bool Changed = false;
+  for (GlobalVariable &GV : MF.getFunction().getParent()->globals()) {
+    if (GV.isDeclarationForLinker())
+      continue;
+    // DataProfileCounts accumulates data profile count across all machine
+    // function instructions, and it can't model the indirect accesses through
+    // other global variables' initializers.
+    // TODO: Analyze the users of module-internal global variables and see
+    // through the users' initializers. Do not place a global variable into
+    // unlikely section if any of its users are potentially hot.
+    auto Iter = DataProfileCounts.find(&GV);
+    if (Iter == DataProfileCounts.end())
+      continue;
+
+    // StaticDataSplitter is made a machine function pass rather than a module
+    // pass because (Lazy)MachineBlockFrequencyInfo is a machine-function
+    // analysis pass and cannot be used for a legacy module pass.
+    // As a result, we use `DataProfileCounts` to accumulate data
+    // profile count across machine functions and update global variable section
+    // prefix once per machine function.
+    // FIXME: Make StaticDataSplitter a module pass under new pass manager
+    // framework, and set global variable section prefix once per module after
+    // analyzing all machine functions.
+    if (PSI->isColdCount(Iter->second.getZExtValue())) {
+      Changed |= GV.updateSectionPrefix("unlikely",
----------------
mingmingl-llvm wrote:

My understanding is that functions have `.hot` or `.unlikely` prefixes when they are placed as a whole (in the [codegenprepare](https://github.com/llvm/llvm-project/blob/69ccb1357fa6cf72063c737d06d6b29ffc465bee/llvm/lib/CodeGen/CodeGenPrepare.cpp#L598) pass), and have `.split` as a prefix when basic blocks of a function are further categorized into hot and non-hot ones (in the [MFS](https://github.com/llvm/llvm-project/blob/69ccb1357fa6cf72063c737d06d6b29ffc465bee/llvm/lib/CodeGen/MachineFunctionSplitter.cpp#L12-L14) pass you mentioned).

Here the granularity is a piece of global variable, and `.split` would be a good name if a global variable itself is split into hot and cold ones. For instance, there should be hot and cold functions in a vtable, but it'd take substantial work to implement an ABI to support the split vtable entries.

https://github.com/llvm/llvm-project/pull/125756