[llvm] [BFI] Solve irreducible SCCs instead of splitting their headers (PR #215170)
Fangrui Song via llvm-commits
llvm-commits at lists.llvm.org
Sun Aug 9 23:11:20 PDT 2026
https://github.com/MaskRay updated https://github.com/llvm/llvm-project/pull/215170
>From 8cfa786d6d59f20fe77780d9ebec1e826592d234 Mon Sep 17 00:00:00 2001
From: Fangrui Song <i at maskray.me>
Date: Sun, 9 Aug 2026 15:35:16 -0700
Subject: [PATCH] [BFI] Solve irreducible SCCs instead of splitting their
headers
In 2014, c5a3139ebd0d approximated irreducible control flow by modelling
an SCC as a loop with multiple headers. http://reviews.llvm.org/D10348
re-distributes the loop mass across the headers in proportion to the
backedge mass each one received -- one step of the same power iteration,
from an assumed split.
Delete that mechanism, package the SCC with a single representative and
solve it instead. `solveIrreducibleMass` iterates the SCC's internal
chain towards its dominant eigenvector and reads the member masses, the
exits and the circulating mass off that, so the entries' relative
frequencies come out of the solve. Power iteration rather than a
relaxation of `f = e + f*P`: the mass `e` entering the SCC is unknown
here, so there is no fixed point to relax towards, only a direction.
Relative error against an exact rationale solve has improved.
```
higher is better | relative error (lower is better)
exact within 1% within 10% | p50 p90 max
before 93 111 118 | 0.99 5.27 521.6
after 93 112 129 | 0.77 2.40 46.2
```
Among the lit tests, selfloops and unequalrows reach their exact ratios,
and nonentry's worst block is off by 1.9x rather than 31.5x.
Bound the iteration rather than run it to convergence: a periodic SCC
oscillates and never terminates, and over that corpus the error
distribution is the same at 16 iterations as at 1000, moving
non-monotonically in between.
This does not subsume -use-iterative-bfi-inference (0a0800c4d10c), which
relaxes the same equations over the whole function, where the entry mass
is known. On yyparse_1 in profile-correlation-irreducible-loops.ll the
solve alone puts b2 at 2.1x the entry against an exact 586.19x, and
inference reaches 586.19x either way. Over the corpus above the two
stack: enabling inference takes within 10% from 118 to 137 functions
before this patch and from 129 to 144 after.
CFGMST weights edges by BFI, so the instrumentation counter indices
permute. Re-record Inputs/irreducible{,_entry}.proftext under the new
numbering; the block counts the tests check are unchanged.
---
.../llvm/Analysis/BlockFrequencyInfoImpl.h | 310 +++++++++---------
llvm/lib/Analysis/BlockFrequencyInfoImpl.cpp | 114 ++-----
.../BlockFrequencyInfo/irreducible.ll | 26 +-
.../BlockFrequencyInfo/irreducible_pgo.ll | 27 +-
.../AArch64/late-taildup-computed-goto.ll | 38 +--
.../basic-block-address-map-pgo-features.ll | 10 +-
llvm/test/CodeGen/X86/fold-loop-of-urem.ll | 19 +-
llvm/test/CodeGen/X86/pr38795.ll | 36 +-
.../PGOProfile/Inputs/irreducible.proftext | 6 +-
.../Inputs/irreducible_entry.proftext | 4 +-
.../profile-correlation-irreducible-loops.ll | 3 +-
11 files changed, 274 insertions(+), 319 deletions(-)
diff --git a/llvm/include/llvm/Analysis/BlockFrequencyInfoImpl.h b/llvm/include/llvm/Analysis/BlockFrequencyInfoImpl.h
index a056c59f3d48f..050cccb7709a1 100644
--- a/llvm/include/llvm/Analysis/BlockFrequencyInfoImpl.h
+++ b/llvm/include/llvm/Analysis/BlockFrequencyInfoImpl.h
@@ -16,7 +16,6 @@
#include "llvm/ADT/BitVector.h"
#include "llvm/ADT/DenseMap.h"
-#include "llvm/ADT/DenseSet.h"
#include "llvm/ADT/GenericCycleInfo.h"
#include "llvm/ADT/GraphTraits.h"
#include "llvm/ADT/PostOrderIterator.h"
@@ -48,6 +47,7 @@
#include <optional>
#include <queue>
#include <string>
+#include <tuple>
#include <utility>
#include <vector>
@@ -221,61 +221,34 @@ class LLVM_ABI BlockFrequencyInfoImplBase {
struct LoopData {
using ExitMap = SmallVector<std::pair<BlockNode, BlockMass>, 4>;
using NodeList = SmallVector<BlockNode, 4>;
- using HeaderMassList = SmallVector<BlockMass, 1>;
- LoopData *Parent; ///< The parent loop.
- bool IsPackaged = false; ///< Whether this has been packaged.
+ LoopData *Parent; ///< The parent loop.
+ bool IsPackaged = false; ///< Whether this has been packaged.
// Has an irreducible SCC in its own nodes; sub-loops package theirs first.
bool ContainsIrreducible = false;
- // Headers are Nodes[0, NumHeaders), sorted. For an irreducible loop, the
- // SCC's entries and the nodes a retreating edge from a non-entry reaches.
- uint32_t NumHeaders = 1;
- ExitMap Exits; ///< Successor edges (and weights).
- NodeList Nodes; ///< Header and the members of the loop.
- HeaderMassList BackedgeMass; ///< Mass returned to each loop header.
+ // A multi-entry SCC rather than a natural loop.
+ bool IsIrreducible = false;
+ ExitMap Exits; ///< Successor edges (and weights).
+ NodeList Nodes; ///< Header and the members of the loop.
+ BlockMass BackedgeMass; ///< Mass that circulates, not exits.
BlockMass Mass;
Scaled64 Scale;
LoopData(LoopData *Parent, const BlockNode &Header)
- : Parent(Parent), Nodes(1, Header), BackedgeMass(1) {}
+ : Parent(Parent), Nodes(1, Header) {}
- template <class It>
- LoopData(LoopData *Parent, It FirstHeader, It LastHeader)
- : Parent(Parent), Nodes(FirstHeader, LastHeader) {
- NumHeaders = Nodes.size();
- BackedgeMass.resize(NumHeaders);
- }
+ /// An irreducible SCC. Its entries are equivalent as far as the enclosing
+ /// region is concerned, so the lowest-RPO member stands for the package
+ /// and solveIrreducibleMass distributes mass among them all.
+ LoopData(LoopData *Parent, NodeList &&Members)
+ : Parent(Parent), IsIrreducible(true), Nodes(std::move(Members)) {}
- template <class It1, class It2>
- LoopData(LoopData *Parent, It1 FirstHeader, It1 LastHeader, It2 FirstOther,
- It2 LastOther)
- : Parent(Parent), Nodes(FirstHeader, LastHeader) {
- NumHeaders = Nodes.size();
- Nodes.insert(Nodes.end(), FirstOther, LastOther);
- BackedgeMass.resize(NumHeaders);
- }
-
- bool isHeader(const BlockNode &Node) const {
- if (isIrreducible())
- return std::binary_search(Nodes.begin(), Nodes.begin() + NumHeaders,
- Node);
- return Node == Nodes[0];
- }
+ bool isHeader(const BlockNode &Node) const { return Node == Nodes[0]; }
BlockNode getHeader() const { return Nodes[0]; }
- bool isIrreducible() const { return NumHeaders > 1; }
-
- HeaderMassList::difference_type getHeaderIndex(const BlockNode &B) {
- assert(isHeader(B) && "this is only valid on loop header blocks");
- if (isIrreducible())
- return std::lower_bound(Nodes.begin(), Nodes.begin() + NumHeaders, B) -
- Nodes.begin();
- return 0;
- }
+ bool isIrreducible() const { return IsIrreducible; }
- NodeList::const_iterator members_begin() const {
- return Nodes.begin() + NumHeaders;
- }
+ NodeList::const_iterator members_begin() const { return Nodes.begin() + 1; }
NodeList::const_iterator members_end() const { return Nodes.end(); }
iterator_range<NodeList::const_iterator> members() const {
@@ -295,8 +268,8 @@ class LLVM_ABI BlockFrequencyInfoImplBase {
/// The innermost loop containing Node that Node does not head.
///
- /// A block can head several nested loops: createIrreducibleLoop() reuses
- /// an SCC's entry blocks as the irreducible loop's headers.
+ /// A block can head several nested loops: an irreducible SCC's
+ /// representative may also head a sub-loop.
LoopData *getContainingLoop() const {
LoopData *L = Loop;
while (L && L->isHeader(Node))
@@ -481,18 +454,6 @@ class LLVM_ABI BlockFrequencyInfoImplBase {
/// Compute the loop scale for a loop.
void computeLoopScale(LoopData &Loop);
- /// Adjust the mass of all headers in an irreducible loop.
- ///
- /// Initially, irreducible loops are assumed to distribute their mass
- /// equally among its headers. This can lead to wrong frequency estimates
- /// since some headers may be executed more frequently than others.
- ///
- /// This adjusts header mass distribution so it matches the weights of
- /// the backedges going into each of the loop headers.
- void adjustLoopHeaderMass(LoopData &Loop);
-
- void distributeIrrLoopHeaderMass(Distribution &Dist);
-
/// Package up a loop.
void packageLoop(LoopData &Loop);
@@ -688,15 +649,10 @@ void IrreducibleGraph::addEdges(const BlockNode &Node,
///
/// In addition to loops, this algorithm has limited support for irreducible
/// SCCs, which are SCCs with multiple entry blocks. Irreducible SCCs are
-/// found from CycleInfo before any mass is distributed, and modelled as loops
-/// with multiple headers.
-///
-/// The headers of irreducible sub-SCCs consist of its entry blocks and all
-/// nodes that are targets of a backedge within it (excluding backedges within
-/// true sub-loops). Block frequency calculations act as if a block is
-/// inserted that intercepts all the edges to the headers. All backedges and
-/// entries point to this block. Its successors are the headers, which split
-/// the frequency evenly.
+/// found from CycleInfo before any mass is distributed, and packaged like a
+/// loop, with the lowest-RPO member standing for the package. There is no
+/// header to sweep from, so \a solveIrreducibleMass() distributes mass among
+/// the members by power iteration instead.
///
/// This algorithm leverages BlockMass and ScaledNumber to maintain precision,
/// separates mass distribution from loop scaling, and dithers to eliminate
@@ -759,16 +715,9 @@ void IrreducibleGraph::addEdges(const BlockNode &Node,
/// - Distribute the mass accordingly, dithering to minimize mass loss,
/// as described in \a distributeMass().
///
-/// In the case of irreducible loops, instead of a single loop header,
-/// there will be several. The computation of backedge masses is similar
-/// but instead of having a single backedge mass, there will be one
-/// backedge per loop header. In these cases, each backedge will carry
-/// a mass proportional to the edge weights along the corresponding
-/// path.
-///
-/// At the end of propagation, the full mass assigned to the loop will be
-/// distributed among the loop headers proportionally according to the
-/// mass flowing through their backedges.
+/// An irreducible SCC is not swept. \a solveIrreducibleMass() iterates
+/// the SCC's internal chain towards its dominant eigenvector and reads the
+/// member masses, the exits and the circulating mass off that.
///
/// Finally, calculate the loop scale from the accumulated backedge mass.
///
@@ -795,37 +744,13 @@ void IrreducibleGraph::addEdges(const BlockNode &Node,
///
/// - The model of irreducible control flow is a rough approximation.
///
-/// Modelling irreducible control flow exactly involves setting up and
-/// solving a group of infinite geometric series. Such precision is
-/// unlikely to be worthwhile, since most of our algorithms give up on
-/// irreducible control flow anyway.
-///
-/// Nevertheless, we might find that we need to get closer. Here's a sort
-/// of TODO list for the model with diminishing returns, to be completed as
-/// necessary.
-///
-/// - The headers for the \a LoopData representing an irreducible SCC
-/// include non-entry blocks. When these extra blocks exist, they
-/// indicate a self-contained irreducible sub-SCC. We could treat them
-/// as sub-loops, rather than arbitrarily shoving the problematic
-/// blocks into the headers of the main irreducible SCC.
-///
-/// - Entry frequencies are assumed to be evenly split between the
-/// headers of a given irreducible SCC, which is the only option if we
-/// need to compute mass in the SCC before its parent loop. Instead,
-/// we could partially compute mass in the parent loop, and stop when
-/// we get to the SCC. Here, we have the correct ratio of entry
-/// masses, which we can use to adjust their relative frequencies.
-/// Compute mass in the SCC, and then continue propagation in the
-/// parent.
-///
-/// - We can propagate mass iteratively through the SCC, for some fixed
-/// number of iterations. Each iteration starts by assigning the entry
-/// blocks their backedge mass from the prior iteration. The final
-/// mass for each block (and each exit, and the total backedge mass
-/// used for computing loop scale) is the sum of all iterations.
-/// (Running this until fixed point would "solve" the geometric
-/// series by simulation.)
+/// \a solveIrreducibleMass() settles an SCC's internal chain, but the mass
+/// entering each entry is unknown until the parent loop is distributed, so
+/// it aims at the quasi-stationary vector rather than the true occupancy.
+/// To get closer, partially compute mass in the parent loop and stop at
+/// the SCC: that gives the correct ratio of entry masses to adjust their
+/// relative frequencies with. Compute mass in the SCC, then continue
+/// propagation in the parent.
template <class BT> class BlockFrequencyInfoImpl : BlockFrequencyInfoImplBase {
using BlockT = typename bfi_detail::TypeMap<BT>::BlockT;
using FunctionT = typename bfi_detail::TypeMap<BT>::FunctionT;
@@ -885,10 +810,11 @@ template <class BT> class BlockFrequencyInfoImpl : BlockFrequencyInfoImplBase {
/// and \a computeIrreducibleMass() for \c Loop if it contains irreducible
/// control flow.
void computeMassInLoop(LoopData &Loop);
+ void solveIrreducibleMass(LoopData &Loop);
- /// Distribute mass in a multi-header loop, seeding the headers from
- /// irr_loop_header_weight metadata and marking them in IsIrrLoopHeader.
- void computeMassInIrreducibleLoop(LoopData &Loop);
+ /// Collect \c Node's successors, resolved through any package, with weights.
+ void getSuccWeights(const BlockNode &Node,
+ SmallVectorImpl<std::pair<BlockNode, uint64_t>> &Out);
/// Compute mass in (and package up) irreducible SCCs.
///
@@ -1205,7 +1131,7 @@ void BlockFrequencyInfoImpl<BT>::computeMassInLoop(LoopData &Loop) {
if (Loop.isIrreducible()) {
LLVM_DEBUG(dbgs() << "isIrreducible = true\n");
- computeMassInIrreducibleLoop(Loop);
+ solveIrreducibleMass(Loop);
} else {
Working[Loop.getHeader().Index].getMass() = BlockMass::getFull();
propagateMassToSuccessors(&Loop, Loop.getHeader());
@@ -1218,60 +1144,119 @@ void BlockFrequencyInfoImpl<BT>::computeMassInLoop(LoopData &Loop) {
}
template <class BT>
-void BlockFrequencyInfoImpl<BT>::computeMassInIrreducibleLoop(LoopData &Loop) {
- Distribution Dist;
- unsigned NumHeadersWithWeight = 0;
- std::optional<uint64_t> MinHeaderWeight;
- DenseSet<uint32_t> HeadersWithoutWeight;
- HeadersWithoutWeight.reserve(Loop.NumHeaders);
- for (uint32_t H = 0; H < Loop.NumHeaders; ++H) {
- auto &HeaderNode = Loop.Nodes[H];
- const BlockT *Block = getBlock(HeaderNode);
- IsIrrLoopHeader.set(Loop.Nodes[H].Index);
- std::optional<uint64_t> HeaderWeight = Block->getIrrLoopHeaderWeight();
- if (!HeaderWeight) {
- LLVM_DEBUG(dbgs() << "Missing irr loop header metadata on "
- << getBlockName(HeaderNode) << "\n");
- HeadersWithoutWeight.insert(H);
+void BlockFrequencyInfoImpl<BT>::getSuccWeights(
+ const BlockNode &Node,
+ SmallVectorImpl<std::pair<BlockNode, uint64_t>> &Out) {
+ Out.clear();
+ if (auto *L = Working[Node.Index].getPackagedLoop()) {
+ for (const auto &E : L->Exits)
+ Out.emplace_back(Working[E.first.Index].getResolvedNode(),
+ E.second.getMass());
+ return;
+ }
+ const BlockT *BB = getBlock(Node);
+ for (auto It : enumerate(children<const BlockT *>(BB))) {
+ BlockNode Succ = getNode(It.value());
+ if (!Succ.isValid())
continue;
+ uint64_t W =
+ getWeightFromBranchProb(BPI->getEdgeProbability(BB, It.index()));
+ Out.emplace_back(Working[Succ.Index].getResolvedNode(),
+ std::max<uint64_t>(1, W));
+ }
+}
+
+// Distribute an irreducible SCC's mass among its members, and record the
+// exits and circulating mass computeLoopScale() needs. For the transition
+// matrix restricted to SCC members, use power iteration to find an approximate
+// solution.
+template <class BT>
+void BlockFrequencyInfoImpl<BT>::solveIrreducibleMass(LoopData &Loop) {
+ const size_t N = Loop.Nodes.size();
+ // Intra-SCC edges (src, dst) and exit edges (src, target), both in src order.
+ SmallVector<std::tuple<uint32_t, uint32_t, Scaled64>> P;
+ SmallVector<std::tuple<uint32_t, BlockNode, Scaled64>> Ex;
+ SmallVector<std::pair<BlockNode, uint64_t>, 8> Succs;
+ for (size_t I = 0; I != N; ++I) {
+ getSuccWeights(Loop.Nodes[I], Succs);
+ uint64_t Total = llvm::sum_of(llvm::make_second_range(Succs));
+ if (!Total)
+ continue;
+ Scaled64 InvTotal = Scaled64::getInverse(Total);
+ for (const auto &S : Succs) {
+ Scaled64 Pr = Scaled64(S.second, 0) * InvTotal;
+ // createIrreducibleLoop sorted Nodes, so a member's position in the
+ // matrix is where it lands in that list.
+ auto It = llvm::lower_bound(Loop.Nodes, S.first);
+ if (It != Loop.Nodes.end() && *It == S.first)
+ P.emplace_back(I, It - Loop.Nodes.begin(), Pr);
+ else
+ Ex.emplace_back(I, S.first, Pr);
+ }
+ }
+
+ // irr_loop_header_weight is a measured block frequency, so pin the members
+ // that carry one and let the rest settle around them. Weights that are all
+ // zero anchor no scale, so start from a uniform split instead.
+ SmallVector<Scaled64> F(N), G(N);
+ SmallVector<bool> Pinned(N, false);
+ Scaled64 Sum;
+ for (size_t I = 0; I != N; ++I)
+ if (auto W = getBlock(Loop.Nodes[I])->getIrrLoopHeaderWeight()) {
+ F[I] = Scaled64(*W, 0);
+ Pinned[I] = true;
+ Sum += F[I];
}
- LLVM_DEBUG(dbgs() << getBlockName(HeaderNode)
- << " has irr loop header weight " << *HeaderWeight
- << "\n");
- NumHeadersWithWeight++;
- uint64_t HeaderWeightValue = *HeaderWeight;
- if (!MinHeaderWeight || HeaderWeightValue < MinHeaderWeight)
- MinHeaderWeight = HeaderWeightValue;
- if (HeaderWeightValue) {
- Dist.addLocal(HeaderNode, HeaderWeightValue);
+ if (Sum.isZero()) {
+ Pinned.assign(N, false);
+ F.assign(N, Scaled64::getInverse(N));
+ Sum = llvm::sum_of(F, Scaled64::getZero());
+ }
+
+ // A backstop, not a convergence criterion: a periodic SCC never settles.
+ const unsigned MaxIterations = 16;
+ // Mass leaks out of the SCC, so F decays geometrically. Sum tracks the
+ // decay; Ratio divides it out so Delta compares directions, not sizes.
+ for (unsigned It = 0; It != MaxIterations; ++It) {
+ G.assign(N, Scaled64::getZero());
+ for (auto [I, J, Pr] : P)
+ G[J] += F[I] * Pr;
+ Scaled64 New;
+ for (size_t I = 0; I != N; ++I) {
+ if (Pinned[I])
+ G[I] = F[I];
+ New += G[I];
}
+ if (New.isZero())
+ break; // nothing circulates; keep the uniform split
+ Scaled64 Ratio = New / Sum;
+ Scaled64 Delta;
+ for (size_t I = 0; I != N; ++I) {
+ Scaled64 Was = Ratio * F[I];
+ Delta += G[I] >= Was ? G[I] - Was : Was - G[I];
+ F[I] = G[I];
+ }
+ Sum = New;
+ if (Delta < New * Scaled64(1, -32))
+ break;
}
- // As a heuristic, if some headers don't have a weight, give them the
- // minimum weight seen (not to disrupt the existing trends too much by
- // using a weight that's in the general range of the other headers' weights,
- // and the minimum seems to perform better than the average.)
- // FIXME: better update in the passes that drop the header weight.
- // If no headers have a weight, give them even weight (use weight 1).
- if (!MinHeaderWeight)
- MinHeaderWeight = 1;
- for (uint32_t H : HeadersWithoutWeight) {
- auto &HeaderNode = Loop.Nodes[H];
- assert(!getBlock(HeaderNode)->getIrrLoopHeaderWeight() &&
- "Shouldn't have a weight metadata");
- uint64_t MinWeight = *MinHeaderWeight;
- LLVM_DEBUG(dbgs() << "Giving weight " << MinWeight << " to "
- << getBlockName(HeaderNode) << "\n");
- if (MinWeight)
- Dist.addLocal(HeaderNode, MinWeight);
+
+ if (!Sum.isZero())
+ for (auto &X : F)
+ X = X / Sum;
+
+ for (size_t I = 0; I != N; ++I)
+ Working[Loop.Nodes[I].Index].getMass() = BlockMass(F[I].scale(UINT64_MAX));
+
+ BlockMass TotalExit;
+ for (auto [I, Succ, Pr] : Ex) {
+ uint64_t M = (F[I] * Pr).scale(UINT64_MAX);
+ if (!M)
+ continue;
+ Loop.Exits.emplace_back(Succ, BlockMass(M));
+ TotalExit += BlockMass(M);
}
- distributeIrrLoopHeaderMass(Dist);
- // Seeded headers are ordered first. Any retreating edge from a non-header
- // targets a header.
- for (const BlockNode &M : Loop.Nodes)
- propagateMassToSuccessors(&Loop, M);
- if (NumHeadersWithWeight == 0)
- // No headers have a metadata. Adjust header mass.
- adjustLoopHeaderMass(Loop);
+ Loop.BackedgeMass = BlockMass::getFull() - TotalExit;
}
template <class BT> void BlockFrequencyInfoImpl<BT>::computeMassInFunction() {
@@ -1601,8 +1586,7 @@ void BlockFrequencyInfoImpl<BT>::computeIrreducibleMass(
// Drop the nodes the new packages absorbed.
assert(OuterLoop->Exits.empty() && "unexpected exits before distribution");
- assert(llvm::all_of(OuterLoop->BackedgeMass,
- [](BlockMass M) { return M.isEmpty(); }) &&
+ assert(OuterLoop->BackedgeMass.isEmpty() &&
"unexpected backedge mass before distribution");
auto O = OuterLoop->Nodes.begin() + 1;
for (auto I = O, E = OuterLoop->Nodes.end(); I != E; ++I)
diff --git a/llvm/lib/Analysis/BlockFrequencyInfoImpl.cpp b/llvm/lib/Analysis/BlockFrequencyInfoImpl.cpp
index fee82843e87dc..27c52afc3baab 100644
--- a/llvm/lib/Analysis/BlockFrequencyInfoImpl.cpp
+++ b/llvm/lib/Analysis/BlockFrequencyInfoImpl.cpp
@@ -350,12 +350,10 @@ void BlockFrequencyInfoImplBase::addToDist(Distribution &Dist,
return;
}
- if (Resolved < Pred) {
- // Every irreducible SCC is packaged before mass distribution, so this is
- // a false backedge from a secondary header of an irreducible OuterLoop.
- assert(isLoopHeader(Pred) && OuterLoop->isIrreducible() &&
- "unhandled irreducible control flow");
- }
+ // Every irreducible SCC is packaged before mass distribution and an
+ // irreducible package is solved rather than swept, so the only retreating
+ // edge left is the one to OuterLoop's header, handled above.
+ assert(Resolved >= Pred && "unhandled irreducible control flow");
LLVM_DEBUG(debugSuccessor(" local "));
Dist.addLocal(Resolved, Weight);
@@ -386,10 +384,7 @@ void BlockFrequencyInfoImplBase::computeLoopScale(LoopData &Loop) {
// LoopScale == 1 / ExitMass
// ExitMass == HeadMass - BackedgeMass
- BlockMass TotalBackedgeMass;
- for (auto &Mass : Loop.BackedgeMass)
- TotalBackedgeMass += Mass;
- BlockMass ExitMass = BlockMass::getFull() - TotalBackedgeMass;
+ BlockMass ExitMass = BlockMass::getFull() - Loop.BackedgeMass;
// Block scale stores the inverse of the scale. If this is an infinite loop,
// its exit mass will be zero. In this case, use an arbitrary scale for the
@@ -398,7 +393,7 @@ void BlockFrequencyInfoImplBase::computeLoopScale(LoopData &Loop) {
ExitMass.isEmpty() ? InfiniteLoopScale : ExitMass.toScaled().inverse();
LLVM_DEBUG(dbgs() << " - exit-mass = " << ExitMass << " ("
- << BlockMass::getFull() << " - " << TotalBackedgeMass
+ << BlockMass::getFull() << " - " << Loop.BackedgeMass
<< ")\n"
<< " - scale = " << Loop.Scale << "\n");
}
@@ -452,7 +447,7 @@ void BlockFrequencyInfoImplBase::distributeMass(const BlockNode &Source,
// Check for a backedge.
if (W.Type == Weight::Backedge) {
- OuterLoop->BackedgeMass[OuterLoop->getHeaderIndex(W.TargetNode)] += Taken;
+ OuterLoop->BackedgeMass += Taken;
LLVM_DEBUG(debugAssign(*this, D, W.TargetNode, Taken, "back"));
continue;
}
@@ -662,8 +657,7 @@ template <> struct GraphTraits<IrreducibleGraph> {
} // end namespace llvm
-/// Package \c SCC into a loop, headed by the nodes marked in \c IsEntry or
-/// \c Extra.
+/// Package \c SCC into a loop represented by its lowest-RPO member.
static void
createIrreducibleLoop(BlockFrequencyInfoImplBase &BFI,
const IrreducibleGraph &G, LoopData *OuterLoop,
@@ -672,35 +666,24 @@ createIrreducibleLoop(BlockFrequencyInfoImplBase &BFI,
const BitVector &IsEntry, const BitVector &Extra) {
LLVM_DEBUG(dbgs() << " - found-scc\n");
- LoopData::NodeList Headers;
- LoopData::NodeList Others;
- for (const auto *I : SCC)
- if (IsEntry.test(G.getIndex(I))) {
- Headers.push_back(I->Node);
- LLVM_DEBUG(dbgs() << " => entry = " << BFI.getBlockName(I->Node)
- << "\n");
- }
- assert(Headers.size() >= 2 &&
- "Expected irreducible CFG; -loop-info is likely invalid");
-
+ // One representative, not a header set: solving the SCC makes the entries'
+ // relative frequencies fall out of the solve rather than out of an assumed
+ // split. Take the lowest RPO node so the choice is deterministic.
+ LoopData::NodeList Members;
+ Members.reserve(SCC.size());
for (const auto *I : SCC) {
- if (IsEntry.test(G.getIndex(I)))
- continue;
- if (Extra.test(G.getIndex(I))) {
- Headers.push_back(I->Node);
- LLVM_DEBUG(dbgs() << " => extra = " << BFI.getBlockName(I->Node)
- << "\n");
- } else {
- Others.push_back(I->Node);
- LLVM_DEBUG(dbgs() << " => other = " << BFI.getBlockName(I->Node)
- << "\n");
- }
+ Members.push_back(I->Node);
+ // The package no longer distinguishes headers; the marking stays because
+ // PGOInstrumentation places counters on isIrrLoopHeader().
+ bool Header = IsEntry.test(G.getIndex(I)) || Extra.test(G.getIndex(I));
+ if (Header)
+ BFI.IsIrrLoopHeader.set(I->Node.Index);
+ LLVM_DEBUG(dbgs() << (Header ? " => header = " : " => member = ")
+ << BFI.getBlockName(I->Node) << "\n");
}
- llvm::sort(Headers);
- llvm::sort(Others);
+ llvm::sort(Members);
- auto Loop = BFI.Loops.emplace(Insert, OuterLoop, Headers.begin(),
- Headers.end(), Others.begin(), Others.end());
+ auto Loop = BFI.Loops.emplace(Insert, OuterLoop, std::move(Members));
// Update loop hierarchy.
for (const auto &N : Loop->Nodes)
@@ -731,7 +714,8 @@ BlockFrequencyInfoImplBase::analyzeIrreducible(
// A node is an entry if an edge from another SCC reaches it, and an extra
// header if a backedge within its SCC targets it. Backedges from entries
- // can have inverted ordering, so they do not make a header.
+ // can have inverted ordering, so they do not make a header. Mass no longer
+ // depends on this split; it only decides isIrrLoopHeader().
BitVector IsEntry(G.Nodes.size());
BitVector Extra(G.Nodes.size());
for (const auto &U : G.Nodes)
@@ -753,51 +737,3 @@ BlockFrequencyInfoImplBase::analyzeIrreducible(
return make_range(std::next(Prev), Insert);
return make_range(Loops.begin(), Insert);
}
-
-void BlockFrequencyInfoImplBase::adjustLoopHeaderMass(LoopData &Loop) {
- assert(Loop.isIrreducible() && "this only makes sense on irreducible loops");
-
- // Since the loop has more than one header block, the mass flowing back into
- // each header will be different. Adjust the mass in each header loop to
- // reflect the masses flowing through back edges.
- //
- // To do this, we distribute the initial mass using the backedge masses
- // as weights for the distribution.
- BlockMass LoopMass = BlockMass::getFull();
- Distribution Dist;
-
- LLVM_DEBUG(dbgs() << "adjust-loop-header-mass:\n");
- for (uint32_t H = 0; H < Loop.NumHeaders; ++H) {
- auto &HeaderNode = Loop.Nodes[H];
- auto &BackedgeMass = Loop.BackedgeMass[H];
- LLVM_DEBUG(dbgs() << " - Add back edge mass for node "
- << getBlockName(HeaderNode) << ": " << BackedgeMass
- << "\n");
- if (BackedgeMass.getMass() > 0)
- Dist.addLocal(HeaderNode, BackedgeMass.getMass());
- else
- LLVM_DEBUG(dbgs() << " Nothing added. Back edge mass is zero\n");
- }
-
- DitheringDistributer D(Dist, LoopMass);
-
- LLVM_DEBUG(dbgs() << " Distribute loop mass " << LoopMass
- << " to headers using above weights\n");
- for (const Weight &W : Dist.Weights) {
- BlockMass Taken = D.takeMass(W.Amount);
- assert(W.Type == Weight::Local && "all weights should be local");
- Working[W.TargetNode.Index].getMass() = Taken;
- LLVM_DEBUG(debugAssign(*this, D, W.TargetNode, Taken, nullptr));
- }
-}
-
-void BlockFrequencyInfoImplBase::distributeIrrLoopHeaderMass(Distribution &Dist) {
- BlockMass LoopMass = BlockMass::getFull();
- DitheringDistributer D(Dist, LoopMass);
- for (const Weight &W : Dist.Weights) {
- BlockMass Taken = D.takeMass(W.Amount);
- assert(W.Type == Weight::Local && "all weights should be local");
- Working[W.TargetNode.Index].getMass() = Taken;
- LLVM_DEBUG(debugAssign(*this, D, W.TargetNode, Taken, nullptr));
- }
-}
diff --git a/llvm/test/Analysis/BlockFrequencyInfo/irreducible.ll b/llvm/test/Analysis/BlockFrequencyInfo/irreducible.ll
index 554053220890f..da52da4bb1213 100644
--- a/llvm/test/Analysis/BlockFrequencyInfo/irreducible.ll
+++ b/llvm/test/Analysis/BlockFrequencyInfo/irreducible.ll
@@ -392,21 +392,21 @@ entry:
br i1 %x, label %left, label %right, !prof !21
left:
-; CHECK-NEXT: left: float = 0.14
+; CHECK-NEXT: left: float = 0.15
br i1 %x, label %top, label %bottom, !prof !22
right:
-; CHECK-NEXT: right: float = 0.42
+; CHECK-NEXT: right: float = 0.45
br i1 %x, label %top, label %bottom, !prof !22
top:
-; CHECK-NEXT: top: float = 8.43
+; CHECK-NEXT: top: float = 3.0
switch i2 %y, label %exit [ i2 0, label %left
i2 1, label %right
i2 2, label %bottom ], !prof !23
bottom:
-; CHECK-NEXT: bottom: float = 4.5,
+; CHECK-NEXT: bottom: float = 2.1698,
br label %top
exit:
@@ -452,17 +452,17 @@ entry:
i32 2, label %c ]
a:
-; CHECK-NEXT: a: float = 1774.9,
+; CHECK-NEXT: a: float = 1820.4,
switch i32 %x, label %a [ i32 1, label %b
i32 2, label %c ], !prof !31
b:
-; CHECK-NEXT: b: float = 1160.5,
+; CHECK-NEXT: b: float = 1137.8,
switch i32 %x, label %a [ i32 1, label %b
i32 2, label %c ], !prof !32
c:
-; CHECK-NEXT: c: float = 1160.5,
+; CHECK-NEXT: c: float = 1137.8,
switch i32 %x, label %a [ i32 1, label %b
i32 2, label %c ], !prof !33
}
@@ -480,17 +480,17 @@ entry:
i32 2, label %o2 ]
lh:
-; CHECK-NEXT: lh: float = 2730.7,
+; CHECK-NEXT: lh: float = 2340.6,
switch i32 %x, label %lh [ i32 1, label %o1
i32 2, label %o2 ], !prof !34
o1:
-; CHECK-NEXT: o1: float = 682.67,
+; CHECK-NEXT: o1: float = 877.71,
switch i32 %x, label %lh [ i32 1, label %o1
i32 2, label %o2 ], !prof !35
o2:
-; CHECK-NEXT: o2: float = 682.67,
+; CHECK-NEXT: o2: float = 877.71,
switch i32 %x, label %lh [ i32 1, label %o1
i32 2, label %o2 ], !prof !35
}
@@ -507,15 +507,15 @@ entry:
br i1 %x, label %a, label %b
a:
-; CHECK-NEXT: a: float = 62.984,
+; CHECK-NEXT: a: float = 519.62,
br i1 %y, label %c, label %b
c:
-; CHECK-NEXT: c: float = 31.0,
+; CHECK-NEXT: c: float = 992.0,
br label %a
b:
-; CHECK-NEXT: b: float = 1.0159,
+; CHECK-NEXT: b: float = 32.0,
br i1 %z, label %a, label %exit
exit:
diff --git a/llvm/test/Analysis/BlockFrequencyInfo/irreducible_pgo.ll b/llvm/test/Analysis/BlockFrequencyInfo/irreducible_pgo.ll
index dabd2fb30ab21..03f6e6b08570c 100644
--- a/llvm/test/Analysis/BlockFrequencyInfo/irreducible_pgo.ll
+++ b/llvm/test/Analysis/BlockFrequencyInfo/irreducible_pgo.ll
@@ -220,6 +220,31 @@ indirectgoto: ; preds = %if.then18, %if.then
; CHECK-LABEL: Printing analysis {{.*}} for function '_Z11irreduciblePh2':
; CHECK: block-frequency-info: _Z11irreduciblePh2
-; CHECK: - sw.bb6: {{.*}} count = 100
+; CHECK: - sw.bb6: {{.*}} count = 300
; CHECK: - sw.bb15: {{.*}} count = 100, irr_loop_header_weight = 100
; CHECK: - indirectgoto: {{.*}} count = 400, irr_loop_header_weight = 400
+
+; Weights of zero pin nothing, so the SCC falls back to the branch weights.
+; The mass still has to leave it.
+define void @all_weights_zero(i1 %x) {
+entry:
+ br i1 %x, label %c1, label %c2, !prof !43
+
+c1:
+ br i1 %x, label %c2, label %exit, !prof !43, !irr_loop !44
+
+c2:
+ br i1 %x, label %c1, label %exit, !prof !43, !irr_loop !44
+
+exit:
+ ret void
+}
+!43 = !{!"branch_weights", i32 3, i32 1}
+!44 = !{!"loop_header_weight", i64 0}
+
+; CHECK-LABEL: Printing analysis {{.*}} for function 'all_weights_zero':
+; CHECK-NEXT: block-frequency-info: all_weights_zero
+; CHECK-NEXT: - entry: float = 1.0, int = [[ENTRY:[0-9]+]]
+; CHECK-NEXT: - c1: float = 2.0,
+; CHECK-NEXT: - c2: float = 2.0,
+; CHECK-NEXT: - exit: float = 1.0, int = [[ENTRY]]
diff --git a/llvm/test/CodeGen/AArch64/late-taildup-computed-goto.ll b/llvm/test/CodeGen/AArch64/late-taildup-computed-goto.ll
index db06a21ad5815..381904f776604 100644
--- a/llvm/test/CodeGen/AArch64/late-taildup-computed-goto.ll
+++ b/llvm/test/CodeGen/AArch64/late-taildup-computed-goto.ll
@@ -42,7 +42,21 @@ define void @test_interp(ptr %frame, ptr %dst) {
; CHECK-NEXT: add x24, x24, #1
; CHECK-NEXT: br x8
; CHECK-NEXT: Ltmp1: ; Block address taken
-; CHECK-NEXT: LBB0_2: ; %op2.bb
+; CHECK-NEXT: LBB0_2: ; %op1.bb
+; CHECK-NEXT: str xzr, [x19]
+; CHECK-NEXT: Ltmp2: ; Block address taken
+; CHECK-NEXT: LBB0_3: ; %op6.bb
+; CHECK-NEXT: ; =>This Inner Loop Header: Depth=1
+; CHECK-NEXT: ldr x0, [x20, #-8]!
+; CHECK-NEXT: ldr x8, [x0, #8]
+; CHECK-NEXT: str x22, [x0]
+; CHECK-NEXT: ldr x8, [x8, #48]
+; CHECK-NEXT: blr x8
+; CHECK-NEXT: add x8, x21, x24, lsl #3
+; CHECK-NEXT: add x24, x24, #1
+; CHECK-NEXT: br x8
+; CHECK-NEXT: Ltmp3: ; Block address taken
+; CHECK-NEXT: LBB0_4: ; %op2.bb
; CHECK-NEXT: ; =>This Inner Loop Header: Depth=1
; CHECK-NEXT: add x8, x21, x24, lsl #3
; CHECK-NEXT: mov x20, xzr
@@ -50,10 +64,10 @@ define void @test_interp(ptr %frame, ptr %dst) {
; CHECK-NEXT: mov x23, xzr
; CHECK-NEXT: add x24, x24, #1
; CHECK-NEXT: br x8
-; CHECK-NEXT: Ltmp2: ; Block address taken
-; CHECK-NEXT: LBB0_3: ; %op4.bb
-; CHECK-NEXT: Ltmp3: ; Block address taken
-; CHECK-NEXT: LBB0_4: ; %op5.bb
+; CHECK-NEXT: Ltmp4: ; Block address taken
+; CHECK-NEXT: LBB0_5: ; %op4.bb
+; CHECK-NEXT: Ltmp5: ; Block address taken
+; CHECK-NEXT: LBB0_6: ; %op5.bb
; CHECK-NEXT: ; =>This Inner Loop Header: Depth=1
; CHECK-NEXT: str x23, [x19]
; CHECK-NEXT: ldur x8, [x23, #12]
@@ -64,20 +78,6 @@ define void @test_interp(ptr %frame, ptr %dst) {
; CHECK-NEXT: add x20, x20, #8
; CHECK-NEXT: add x24, x24, #1
; CHECK-NEXT: br x8
-; CHECK-NEXT: Ltmp4: ; Block address taken
-; CHECK-NEXT: LBB0_5: ; %op1.bb
-; CHECK-NEXT: str xzr, [x19]
-; CHECK-NEXT: Ltmp5: ; Block address taken
-; CHECK-NEXT: LBB0_6: ; %op6.bb
-; CHECK-NEXT: ; =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: ldr x0, [x20, #-8]!
-; CHECK-NEXT: ldr x8, [x0, #8]
-; CHECK-NEXT: str x22, [x0]
-; CHECK-NEXT: ldr x8, [x8, #48]
-; CHECK-NEXT: blr x8
-; CHECK-NEXT: add x8, x21, x24, lsl #3
-; CHECK-NEXT: add x24, x24, #1
-; CHECK-NEXT: br x8
; CHECK-NEXT: .loh AdrpAdd Lloh0, Lloh1
entry:
br label %loop.header
diff --git a/llvm/test/CodeGen/X86/basic-block-address-map-pgo-features.ll b/llvm/test/CodeGen/X86/basic-block-address-map-pgo-features.ll
index 8dff2873b12c4..70f9c37cdeef2 100644
--- a/llvm/test/CodeGen/X86/basic-block-address-map-pgo-features.ll
+++ b/llvm/test/CodeGen/X86/basic-block-address-map-pgo-features.ll
@@ -115,13 +115,13 @@ declare i32 @__gxx_personality_v0(...)
; PGO-NONE-NOT: .ascii "\271\235\376\332\245\200\356\017" # basic block frequency
; PGO-NONE-NOT: .byte 2 # basic block successor count
; PGO-FEC-NEXT: .byte 100 # function entry count
-; PGO-BBF-NEXT: .ascii "\271\235\376\332\245\200\356\017" # basic block frequency
+; PGO-BBF-NEXT: .ascii "\200\344\200\200\262\200\200\035" # basic block frequency
; PGO-BRP-NEXT: .byte 2 # basic block successor count
; PGO-BRP-NEXT: .byte 1 # successor BB ID
; PGO-BRP-NEXT: .ascii "\346\314\231\263\006" # successor branch probability
; PGO-BRP-NEXT: .byte 3 # successor BB ID
; PGO-BRP-NEXT: .ascii "\232\263\346\314\001" # successor branch probability
-; PGO-BBF-NEXT: .ascii "\202\301\341\375\205\200\200\003" # basic block frequency
+; PGO-BBF-NEXT: .ascii "\200\340\200\200\260\200\200\030" # basic block frequency
; PGO-BRP-NEXT: .byte 2 # basic block successor count
; PGO-BRP-NEXT: .byte 3 # successor BB ID
; PGO-BRP-NEXT: .ascii "\200\360\377\377\007" # successor branch probability
@@ -133,15 +133,15 @@ declare i32 @__gxx_personality_v0(...)
; PGO-BRP-NEXT: .ascii "\200\200\200\200\007" # successor branch probability
; PGO-BRP-NEXT: .byte 4 # successor BB ID
; PGO-BRP-NEXT: .ascii "\200\200\200\200\001" # successor branch probability
-; PGO-BBF-NEXT: .ascii "\271\235\376\332\245\200\356\017" # basic block frequency
+; PGO-BBF-NEXT: .ascii "\200\344\200\200\262\200\200\035" # basic block frequency
; PGO-BRP-NEXT: .byte 0 # basic block successor count
-; PGO-BBF-NEXT: .ascii "\210\214\356\257\200\200\230\002" # basic block frequency
+; PGO-BBF-NEXT: .ascii "\200\220\200\200\210\200\200\004" # basic block frequency
; PGO-BRP-NEXT: .byte 2 # basic block successor count
; PGO-BRP-NEXT: .byte 1 # successor BB ID
; PGO-BRP-NEXT: .ascii "\200\200\200\200\006" # successor branch probability
; PGO-BRP-NEXT: .byte 5 # successor BB ID
; PGO-BRP-NEXT: .ascii "\200\200\200\200\002" # successor branch probability
-; PGO-BBF-NEXT: .ascii "\235\323\243\200#" # basic block frequency
+; PGO-BBF-NEXT: .ascii "\333\322\340\374/" # basic block frequency
; PGO-BRP-NEXT: .byte 1 # basic block successor count
; PGO-BRP-NEXT: .byte 5 # successor BB ID
; PGO-BRP-NEXT: .ascii "\200\200\200\200\b" # successor branch probability
diff --git a/llvm/test/CodeGen/X86/fold-loop-of-urem.ll b/llvm/test/CodeGen/X86/fold-loop-of-urem.ll
index cb1c078ee5129..f3b9af4eb08e8 100644
--- a/llvm/test/CodeGen/X86/fold-loop-of-urem.ll
+++ b/llvm/test/CodeGen/X86/fold-loop-of-urem.ll
@@ -973,10 +973,14 @@ define void @simple_urem_fail_bad_loop(i32 %N, i32 %rem_amt) nounwind {
; CHECK-NEXT: jne .LBB16_4
; CHECK-NEXT: # %bb.1:
; CHECK-NEXT: xorl %r14d, %r14d
-; CHECK-NEXT: .LBB16_2: # %for.cond
; CHECK-NEXT: cmpl %ebp, %r14d
-; CHECK-NEXT: jae .LBB16_5
-; CHECK-NEXT: # %bb.3: # %for.body
+; CHECK-NEXT: jb .LBB16_3
+; CHECK-NEXT: .LBB16_5: # %for.end
+; CHECK-NEXT: popq %rbx
+; CHECK-NEXT: popq %r14
+; CHECK-NEXT: popq %rbp
+; CHECK-NEXT: retq
+; CHECK-NEXT: .LBB16_3: # %for.body
; CHECK-NEXT: movl %r14d, %edi
; CHECK-NEXT: xorl $1, %edi
; CHECK-NEXT: callq use.i32 at PLT
@@ -987,12 +991,9 @@ define void @simple_urem_fail_bad_loop(i32 %N, i32 %rem_amt) nounwind {
; CHECK-NEXT: movl %edx, %edi
; CHECK-NEXT: callq use.i32 at PLT
; CHECK-NEXT: incl %r14d
-; CHECK-NEXT: jmp .LBB16_2
-; CHECK-NEXT: .LBB16_5: # %for.end
-; CHECK-NEXT: popq %rbx
-; CHECK-NEXT: popq %r14
-; CHECK-NEXT: popq %rbp
-; CHECK-NEXT: retq
+; CHECK-NEXT: cmpl %ebp, %r14d
+; CHECK-NEXT: jb .LBB16_3
+; CHECK-NEXT: jmp .LBB16_5
entry:
%call = call i32 @get.i32()
%tobool.not = icmp eq i32 %call, 0
diff --git a/llvm/test/CodeGen/X86/pr38795.ll b/llvm/test/CodeGen/X86/pr38795.ll
index 3c23fdafa97e3..cf7ea73145d65 100644
--- a/llvm/test/CodeGen/X86/pr38795.ll
+++ b/llvm/test/CodeGen/X86/pr38795.ll
@@ -25,29 +25,33 @@ define dso_local void @fn() {
; CHECK-NEXT: xorl %ebx, %ebx
; CHECK-NEXT: # implicit-def: $ecx
; CHECK-NEXT: # implicit-def: $edi
-; CHECK-NEXT: # implicit-def: $dl
; CHECK-NEXT: # implicit-def: $al
; CHECK-NEXT: # kill: killed $al
-; CHECK-NEXT: # implicit-def: $ebp
+; CHECK-NEXT: # implicit-def: $al
+; CHECK-NEXT: # kill: killed $al
+; CHECK-NEXT: # implicit-def: $edx
; CHECK-NEXT: jmp .LBB0_1
; CHECK-NEXT: .p2align 4
; CHECK-NEXT: .LBB0_15: # %for.inc
; CHECK-NEXT: # in Loop: Header=BB0_1 Depth=1
; CHECK-NEXT: movzbl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 1-byte Folded Reload
; CHECK-NEXT: movb %al, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Spill
+; CHECK-NEXT: movzbl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 1-byte Folded Reload
+; CHECK-NEXT: movb %al, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Spill
; CHECK-NEXT: .LBB0_1: # %for.cond
; CHECK-NEXT: # =>This Loop Header: Depth=1
; CHECK-NEXT: # Child Loop BB0_19 Depth 2
+; CHECK-NEXT: movl %edx, %ebp
; CHECK-NEXT: testb %bl, %bl
; CHECK-NEXT: jne .LBB0_3
; CHECK-NEXT: # %bb.2: # %if.then
; CHECK-NEXT: # in Loop: Header=BB0_1 Depth=1
-; CHECK-NEXT: movb %dl, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Spill
; CHECK-NEXT: movl $.str, (%esp)
; CHECK-NEXT: calll printf
; CHECK-NEXT: # implicit-def: $eax
; CHECK-NEXT: movzbl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 1-byte Folded Reload
; CHECK-NEXT: movb %cl, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Spill
+; CHECK-NEXT: movl %ebp, %edx
; CHECK-NEXT: jmp .LBB0_5
; CHECK-NEXT: .p2align 4
; CHECK-NEXT: .LBB0_3: # %if.end
@@ -58,25 +62,27 @@ define dso_local void @fn() {
; CHECK-NEXT: movl %eax, %esi
; CHECK-NEXT: movl %ecx, %eax
; CHECK-NEXT: movl $0, h
-; CHECK-NEXT: movzbl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 1-byte Folded Reload
-; CHECK-NEXT: cmpb $8, %dl
+; CHECK-NEXT: movb {{[-0-9]+}}(%e{{[sb]}}p), %ah # 1-byte Reload
+; CHECK-NEXT: cmpb $8, %ah
; CHECK-NEXT: jg .LBB0_7
; CHECK-NEXT: # %bb.4: # %if.then13
; CHECK-NEXT: # in Loop: Header=BB0_1 Depth=1
; CHECK-NEXT: movb %al, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Spill
; CHECK-NEXT: movl $.str, (%esp)
+; CHECK-NEXT: movb %ah, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Spill
; CHECK-NEXT: calll printf
-; CHECK-NEXT: movzbl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 1-byte Folded Reload
; CHECK-NEXT: testb %bl, %bl
; CHECK-NEXT: movl %esi, %ecx
; CHECK-NEXT: # implicit-def: $eax
+; CHECK-NEXT: movzbl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 1-byte Folded Reload
; CHECK-NEXT: movb %dl, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Spill
+; CHECK-NEXT: movl %ebp, %edx
; CHECK-NEXT: jne .LBB0_15
; CHECK-NEXT: jmp .LBB0_5
; CHECK-NEXT: .p2align 4
; CHECK-NEXT: .LBB0_7: # %if.end21
; CHECK-NEXT: # in Loop: Header=BB0_1 Depth=1
-; CHECK-NEXT: # implicit-def: $ebp
+; CHECK-NEXT: # implicit-def: $edx
; CHECK-NEXT: jmp .LBB0_8
; CHECK-NEXT: .p2align 4
; CHECK-NEXT: .LBB0_5: # %for.cond35
@@ -93,13 +99,13 @@ define dso_local void @fn() {
; CHECK-NEXT: # kill: killed $cl
; CHECK-NEXT: # implicit-def: $cl
; CHECK-NEXT: # kill: killed $cl
-; CHECK-NEXT: # implicit-def: $ebp
+; CHECK-NEXT: # implicit-def: $edx
; CHECK-NEXT: jmp .LBB0_5
; CHECK-NEXT: .p2align 4
; CHECK-NEXT: .LBB0_6: # in Loop: Header=BB0_1 Depth=1
; CHECK-NEXT: xorl %edi, %edi
; CHECK-NEXT: movzbl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 1-byte Folded Reload
-; CHECK-NEXT: movzbl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 1-byte Folded Reload
+; CHECK-NEXT: movb {{[-0-9]+}}(%e{{[sb]}}p), %ah # 1-byte Reload
; CHECK-NEXT: jmp .LBB0_19
; CHECK-NEXT: .p2align 4
; CHECK-NEXT: .LBB0_8: # %ae
@@ -110,6 +116,7 @@ define dso_local void @fn() {
; CHECK-NEXT: # implicit-def: $eax
; CHECK-NEXT: testb %bl, %bl
; CHECK-NEXT: jne .LBB0_11
+; CHECK-NEXT: .p2align 4
; CHECK-NEXT: .LBB0_16: # %if.end39
; CHECK-NEXT: # in Loop: Header=BB0_1 Depth=1
; CHECK-NEXT: testl %eax, %eax
@@ -122,9 +129,9 @@ define dso_local void @fn() {
; CHECK-NEXT: calll printf
; CHECK-NEXT: .LBB0_18: # %for.end46
; CHECK-NEXT: # in Loop: Header=BB0_1 Depth=1
-; CHECK-NEXT: # implicit-def: $dl
+; CHECK-NEXT: # implicit-def: $ah
; CHECK-NEXT: # implicit-def: $al
-; CHECK-NEXT: # implicit-def: $ebp
+; CHECK-NEXT: # implicit-def: $edx
; CHECK-NEXT: .p2align 4
; CHECK-NEXT: .LBB0_19: # %for.cond47
; CHECK-NEXT: # Parent Loop BB0_1 Depth=1
@@ -137,16 +144,17 @@ define dso_local void @fn() {
; CHECK-NEXT: # in Loop: Header=BB0_1 Depth=1
; CHECK-NEXT: movb %al, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Spill
; CHECK-NEXT: xorl %ecx, %ecx
-; CHECK-NEXT: testb %dl, %dl
+; CHECK-NEXT: testb %ah, %ah
+; CHECK-NEXT: movb %ah, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Spill
; CHECK-NEXT: je .LBB0_15
; CHECK-NEXT: # %bb.13: # %if.end26
; CHECK-NEXT: # in Loop: Header=BB0_1 Depth=1
-; CHECK-NEXT: testl %ebp, %ebp
+; CHECK-NEXT: testl %edx, %edx
; CHECK-NEXT: jne .LBB0_15
; CHECK-NEXT: # %bb.14: # %if.then31
; CHECK-NEXT: # in Loop: Header=BB0_1 Depth=1
; CHECK-NEXT: xorl %ecx, %ecx
-; CHECK-NEXT: xorl %ebp, %ebp
+; CHECK-NEXT: xorl %edx, %edx
; CHECK-NEXT: jmp .LBB0_15
entry:
br label %for.cond
diff --git a/llvm/test/Transforms/PGOProfile/Inputs/irreducible.proftext b/llvm/test/Transforms/PGOProfile/Inputs/irreducible.proftext
index d0fdac4193553..ab6208a6bafc8 100644
--- a/llvm/test/Transforms/PGOProfile/Inputs/irreducible.proftext
+++ b/llvm/test/Transforms/PGOProfile/Inputs/irreducible.proftext
@@ -5,8 +5,8 @@ _Z11irreducibleii
# Num Counters:
6
# Counter Values:
-1000
950
+1000
373
100
1
@@ -19,11 +19,11 @@ _Z11irreduciblePh
9
# Counter Values:
300
-1
201
+99
+1
1
99
0
0
-99
0
diff --git a/llvm/test/Transforms/PGOProfile/Inputs/irreducible_entry.proftext b/llvm/test/Transforms/PGOProfile/Inputs/irreducible_entry.proftext
index 361a63c8e3e7f..71b4da5a24706 100644
--- a/llvm/test/Transforms/PGOProfile/Inputs/irreducible_entry.proftext
+++ b/llvm/test/Transforms/PGOProfile/Inputs/irreducible_entry.proftext
@@ -7,8 +7,8 @@ _Z11irreducibleii
6
# Counter Values:
1
-1000
950
+1000
373
100
0
@@ -22,9 +22,9 @@ _Z11irreduciblePh
1
300
201
+99
1
99
0
0
-99
0
diff --git a/llvm/test/Transforms/SampleProfile/profile-correlation-irreducible-loops.ll b/llvm/test/Transforms/SampleProfile/profile-correlation-irreducible-loops.ll
index ef2fcc6a9e248..e198cfebacd0a 100644
--- a/llvm/test/Transforms/SampleProfile/profile-correlation-irreducible-loops.ll
+++ b/llvm/test/Transforms/SampleProfile/profile-correlation-irreducible-loops.ll
@@ -75,7 +75,8 @@ b4:
b5:
call void @llvm.pseudoprobe(i64 -7702751003264189226, i64 5, i32 0, i64 -1)
br i1 %cmp, label %b9, label %b6
-; CHECK: - b5: float = {{.*}}, int = {{.*}}, count = 2
+; b5 is exactly 1.5x the entry block, so the printed count is a rounding tie.
+; CHECK: - b5: float = 0.00079554, int = {{.*}}, count = {{[12]}}
b6:
call void @llvm.pseudoprobe(i64 -7702751003264189226, i64 6, i32 0, i64 -1)
More information about the llvm-commits
mailing list