[llvm-branch-commits] [llvm] [Transforms] Refactor CreateControlFlowHub (PR #103013)
via llvm-branch-commits
llvm-branch-commits at lists.llvm.org
Mon Aug 12 23:54:17 PDT 2024
llvmbot wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-llvm-transforms
Author: Sameer Sahasrabuddhe (ssahasra)
<details>
<summary>Changes</summary>
CreateControlFlowHub is a method that redirects control flow edges from a set of incoming blocks to a set of outgoing blocks through a new set of "guard" blocks. This is now refactored into a separate file with one enhancement: The input to the method is now a set of branches rather than two sets of blocks.
The original implementation reroutes every edge from incoming blocks to outgoing blocks. But it is possible that for some incoming block InBB, some successor S might be in the set of outgoing blocks, but that particular edge should not be rerouted. The new implementation makes this possible by allowing the user to specify the targets of each branch that need to be rerouted.
This is needed when improving the implementation of FixIrreducible #<!-- -->101386. Current uses in FixIrreducible and UnifyLoopExits do not demonstrate this finer control over the edges being rerouted.
---
Patch is 53.83 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/103013.diff
13 Files Affected:
- (modified) llvm/include/llvm/Transforms/Utils/BasicBlockUtils.h (-75)
- (added) llvm/include/llvm/Transforms/Utils/ControlFlowUtils.h (+121)
- (modified) llvm/lib/Transforms/Utils/BasicBlockUtils.cpp (-314)
- (modified) llvm/lib/Transforms/Utils/CMakeLists.txt (+1)
- (added) llvm/lib/Transforms/Utils/ControlFlowUtils.cpp (+341)
- (modified) llvm/lib/Transforms/Utils/FixIrreducible.cpp (+25-10)
- (modified) llvm/lib/Transforms/Utils/UnifyLoopExits.cpp (+30-39)
- (modified) llvm/test/CodeGen/AMDGPU/local-atomicrmw-fadd.ll (+16-16)
- (modified) llvm/test/Transforms/FixIrreducible/basic.ll (+2-2)
- (modified) llvm/test/Transforms/FixIrreducible/bug45623.ll (+2-1)
- (modified) llvm/test/Transforms/FixIrreducible/nested.ll (+2-1)
- (modified) llvm/test/Transforms/FixIrreducible/switch.ll (+2-1)
- (modified) llvm/test/Transforms/FixIrreducible/unreachable.ll (+3-1)
``````````diff
diff --git a/llvm/include/llvm/Transforms/Utils/BasicBlockUtils.h b/llvm/include/llvm/Transforms/Utils/BasicBlockUtils.h
index c99df6bf94d025..b447942ffbd676 100644
--- a/llvm/include/llvm/Transforms/Utils/BasicBlockUtils.h
+++ b/llvm/include/llvm/Transforms/Utils/BasicBlockUtils.h
@@ -602,81 +602,6 @@ bool SplitIndirectBrCriticalEdges(Function &F, bool IgnoreBlocksWithoutPHI,
BranchProbabilityInfo *BPI = nullptr,
BlockFrequencyInfo *BFI = nullptr);
-/// Given a set of incoming and outgoing blocks, create a "hub" such that every
-/// edge from an incoming block InBB to an outgoing block OutBB is now split
-/// into two edges, one from InBB to the hub and another from the hub to
-/// OutBB. The hub consists of a series of guard blocks, one for each outgoing
-/// block. Each guard block conditionally branches to the corresponding outgoing
-/// block, or the next guard block in the chain. These guard blocks are returned
-/// in the argument vector.
-///
-/// Since the control flow edges from InBB to OutBB have now been replaced, the
-/// function also updates any PHINodes in OutBB. For each such PHINode, the
-/// operands corresponding to incoming blocks are moved to a new PHINode in the
-/// hub, and the hub is made an operand of the original PHINode.
-///
-/// Input CFG:
-/// ----------
-///
-/// Def
-/// |
-/// v
-/// In1 In2
-/// | |
-/// | |
-/// v v
-/// Foo ---> Out1 Out2
-/// |
-/// v
-/// Use
-///
-///
-/// Create hub: Incoming = {In1, In2}, Outgoing = {Out1, Out2}
-/// ----------------------------------------------------------
-///
-/// Def
-/// |
-/// v
-/// In1 In2 Foo
-/// | Hub | |
-/// | + - - | - - + |
-/// | ' v ' V
-/// +------> Guard1 -----> Out1
-/// ' | '
-/// ' v '
-/// ' Guard2 -----> Out2
-/// ' ' |
-/// + - - - - - + |
-/// v
-/// Use
-///
-/// Limitations:
-/// -----------
-/// 1. This assumes that all terminators in the CFG are direct branches (the
-/// "br" instruction). The presence of any other control flow such as
-/// indirectbr, switch or callbr will cause an assert.
-///
-/// 2. The updates to the PHINodes are not sufficient to restore SSA
-/// form. Consider a definition Def, its use Use, incoming block In2 and
-/// outgoing block Out2, such that:
-/// a. In2 is reachable from D or contains D.
-/// b. U is reachable from Out2 or is contained in Out2.
-/// c. U is not a PHINode if U is contained in Out2.
-///
-/// Clearly, Def dominates Out2 since the program is valid SSA. But when the
-/// hub is introduced, there is a new path through the hub along which Use is
-/// reachable from entry without passing through Def, and SSA is no longer
-/// valid. To fix this, we need to look at all the blocks post-dominated by
-/// the hub on the one hand, and dominated by Out2 on the other. This is left
-/// for the caller to accomplish, since each specific use of this function
-/// may have additional information which simplifies this fixup. For example,
-/// see restoreSSA() in the UnifyLoopExits pass.
-BasicBlock *CreateControlFlowHub(
- DomTreeUpdater *DTU, SmallVectorImpl<BasicBlock *> &GuardBlocks,
- const SetVector<BasicBlock *> &Predecessors,
- const SetVector<BasicBlock *> &Successors, const StringRef Prefix,
- std::optional<unsigned> MaxControlFlowBooleans = std::nullopt);
-
// Utility function for inverting branch condition and for swapping its
// successors
void InvertBranch(BranchInst *PBI, IRBuilderBase &Builder);
diff --git a/llvm/include/llvm/Transforms/Utils/ControlFlowUtils.h b/llvm/include/llvm/Transforms/Utils/ControlFlowUtils.h
new file mode 100644
index 00000000000000..f5d237bc96627f
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Utils/ControlFlowUtils.h
@@ -0,0 +1,121 @@
+//===- Transforms/Utils/ControlFlowUtils.h --------------------*- C++ -*---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Utilities to manipulate the CFG and restore SSA for the new control flow.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TRANSFORMS_UTILS_CONTROLFLOWUTILS_H
+#define LLVM_TRANSFORMS_UTILS_CONTROLFLOWUTILS_H
+
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+
+namespace llvm {
+
+class BasicBlock;
+class DomTreeUpdater;
+
+/// Given a set of branch descriptors [BB, Succ0, Succ1], create a "hub" such
+/// that the control flow from each BB to a successor is now split into two
+/// edges, one from BB to the hub and another from the hub to the successor. The
+/// hub consists of a series of guard blocks, one for each outgoing block. Each
+/// guard block conditionally branches to the corresponding outgoing block, or
+/// the next guard block in the chain. These guard blocks are returned in the
+/// argument vector.
+///
+/// This also updates any PHINodes in the successor. For each such PHINode, the
+/// operands corresponding to incoming blocks are moved to a new PHINode in the
+/// hub, and the hub is made an operand of the original PHINode.
+///
+/// Note that for some block BB with a conditional branch, it is not necessary
+/// that both successors are rerouted. The client specifies this by setting
+/// either Succ0 or Succ1 to nullptr, in which case, the corresponding successor
+/// is not rerouted.
+///
+/// Input CFG:
+/// ----------
+///
+/// Def
+/// |
+/// v
+/// In1 In2
+/// | |
+/// | |
+/// v v
+/// Foo ---> Out1 Out2
+/// |
+/// v
+/// Use
+///
+///
+/// Create hub: Incoming = {In1, In2}, Outgoing = {Out1, Out2}
+/// ----------------------------------------------------------
+///
+/// Def
+/// |
+/// v
+/// In1 In2 Foo
+/// | Hub | |
+/// | + - - | - - + |
+/// | ' v ' V
+/// +------> Guard1 -----> Out1
+/// ' | '
+/// ' v '
+/// ' Guard2 -----> Out2
+/// ' ' |
+/// + - - - - - + |
+/// v
+/// Use
+///
+/// Limitations:
+/// -----------
+/// 1. This assumes that all terminators in the CFG are direct branches (the
+/// "br" instruction). The presence of any other control flow such as
+/// indirectbr, switch or callbr will cause an assert.
+///
+/// 2. The updates to the PHINodes are not sufficient to restore SSA
+/// form. Consider a definition Def, its use Use, incoming block In2 and
+/// outgoing block Out2, such that:
+/// a. In2 is reachable from D or contains D.
+/// b. U is reachable from Out2 or is contained in Out2.
+/// c. U is not a PHINode if U is contained in Out2.
+///
+/// Clearly, Def dominates Out2 since the program is valid SSA. But when the
+/// hub is introduced, there is a new path through the hub along which Use is
+/// reachable from entry without passing through Def, and SSA is no longer
+/// valid. To fix this, we need to look at all the blocks post-dominated by
+/// the hub on the one hand, and dominated by Out2 on the other. This is left
+/// for the caller to accomplish, since each specific use of this function
+/// may have additional information which simplifies this fixup. For example,
+/// see restoreSSA() in the UnifyLoopExits pass.
+struct ControlFlowHub {
+ struct BranchDescriptor {
+ BasicBlock *BB;
+ BasicBlock *Succ0;
+ BasicBlock *Succ1;
+
+ BranchDescriptor(BasicBlock *BB, BasicBlock *Succ0, BasicBlock *Succ1)
+ : BB(BB), Succ0(Succ0), Succ1(Succ1) {}
+ };
+
+ void addBranch(BasicBlock *BB, BasicBlock *Succ0, BasicBlock *Succ1) {
+ Branches.emplace_back(BB, Succ0, Succ1);
+ }
+
+ BasicBlock *
+ finalize(DomTreeUpdater *DTU, SmallVectorImpl<BasicBlock *> &GuardBlocks,
+ const StringRef Prefix,
+ std::optional<unsigned> MaxControlFlowBooleans = std::nullopt);
+
+ SmallVector<BranchDescriptor> Branches;
+};
+
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_UTILS_CONTROLFLOWUTILS_H
diff --git a/llvm/lib/Transforms/Utils/BasicBlockUtils.cpp b/llvm/lib/Transforms/Utils/BasicBlockUtils.cpp
index c78965aaeba3ca..4144c7993b7e42 100644
--- a/llvm/lib/Transforms/Utils/BasicBlockUtils.cpp
+++ b/llvm/lib/Transforms/Utils/BasicBlockUtils.cpp
@@ -1891,320 +1891,6 @@ BranchInst *llvm::GetIfCondition(BasicBlock *BB, BasicBlock *&IfTrue,
return BI;
}
-// After creating a control flow hub, the operands of PHINodes in an outgoing
-// block Out no longer match the predecessors of that block. Predecessors of Out
-// that are incoming blocks to the hub are now replaced by just one edge from
-// the hub. To match this new control flow, the corresponding values from each
-// PHINode must now be moved a new PHINode in the first guard block of the hub.
-//
-// This operation cannot be performed with SSAUpdater, because it involves one
-// new use: If the block Out is in the list of Incoming blocks, then the newly
-// created PHI in the Hub will use itself along that edge from Out to Hub.
-static void reconnectPhis(BasicBlock *Out, BasicBlock *GuardBlock,
- const SetVector<BasicBlock *> &Incoming,
- BasicBlock *FirstGuardBlock) {
- auto I = Out->begin();
- while (I != Out->end() && isa<PHINode>(I)) {
- auto Phi = cast<PHINode>(I);
- auto NewPhi =
- PHINode::Create(Phi->getType(), Incoming.size(),
- Phi->getName() + ".moved", FirstGuardBlock->begin());
- bool AllUndef = true;
- for (auto *In : Incoming) {
- Value *V = UndefValue::get(Phi->getType());
- if (In == Out) {
- V = NewPhi;
- } else if (Phi->getBasicBlockIndex(In) != -1) {
- V = Phi->removeIncomingValue(In, false);
- AllUndef &= isa<UndefValue>(V);
- }
- NewPhi->addIncoming(V, In);
- }
- assert(NewPhi->getNumIncomingValues() == Incoming.size());
- Value *NewV = NewPhi;
- if (AllUndef) {
- NewPhi->eraseFromParent();
- NewV = UndefValue::get(Phi->getType());
- }
- if (Phi->getNumOperands() == 0) {
- Phi->replaceAllUsesWith(NewV);
- I = Phi->eraseFromParent();
- continue;
- }
- Phi->addIncoming(NewV, GuardBlock);
- ++I;
- }
-}
-
-using BBPredicates = DenseMap<BasicBlock *, Instruction *>;
-using BBSetVector = SetVector<BasicBlock *>;
-
-// Redirects the terminator of the incoming block to the first guard
-// block in the hub. The condition of the original terminator (if it
-// was conditional) and its original successors are returned as a
-// tuple <condition, succ0, succ1>. The function additionally filters
-// out successors that are not in the set of outgoing blocks.
-//
-// - condition is non-null iff the branch is conditional.
-// - Succ1 is non-null iff the sole/taken target is an outgoing block.
-// - Succ2 is non-null iff condition is non-null and the fallthrough
-// target is an outgoing block.
-static std::tuple<Value *, BasicBlock *, BasicBlock *>
-redirectToHub(BasicBlock *BB, BasicBlock *FirstGuardBlock,
- const BBSetVector &Outgoing) {
- assert(isa<BranchInst>(BB->getTerminator()) &&
- "Only support branch terminator.");
- auto Branch = cast<BranchInst>(BB->getTerminator());
- auto Condition = Branch->isConditional() ? Branch->getCondition() : nullptr;
-
- BasicBlock *Succ0 = Branch->getSuccessor(0);
- BasicBlock *Succ1 = nullptr;
- Succ0 = Outgoing.count(Succ0) ? Succ0 : nullptr;
-
- if (Branch->isUnconditional()) {
- Branch->setSuccessor(0, FirstGuardBlock);
- assert(Succ0);
- } else {
- Succ1 = Branch->getSuccessor(1);
- Succ1 = Outgoing.count(Succ1) ? Succ1 : nullptr;
- assert(Succ0 || Succ1);
- if (Succ0 && !Succ1) {
- Branch->setSuccessor(0, FirstGuardBlock);
- } else if (Succ1 && !Succ0) {
- Branch->setSuccessor(1, FirstGuardBlock);
- } else {
- Branch->eraseFromParent();
- BranchInst::Create(FirstGuardBlock, BB);
- }
- }
-
- assert(Succ0 || Succ1);
- return std::make_tuple(Condition, Succ0, Succ1);
-}
-// Setup the branch instructions for guard blocks.
-//
-// Each guard block terminates in a conditional branch that transfers
-// control to the corresponding outgoing block or the next guard
-// block. The last guard block has two outgoing blocks as successors
-// since the condition for the final outgoing block is trivially
-// true. So we create one less block (including the first guard block)
-// than the number of outgoing blocks.
-static void setupBranchForGuard(SmallVectorImpl<BasicBlock *> &GuardBlocks,
- const BBSetVector &Outgoing,
- BBPredicates &GuardPredicates) {
- // To help keep the loop simple, temporarily append the last
- // outgoing block to the list of guard blocks.
- GuardBlocks.push_back(Outgoing.back());
-
- for (int i = 0, e = GuardBlocks.size() - 1; i != e; ++i) {
- auto Out = Outgoing[i];
- assert(GuardPredicates.count(Out));
- BranchInst::Create(Out, GuardBlocks[i + 1], GuardPredicates[Out],
- GuardBlocks[i]);
- }
-
- // Remove the last block from the guard list.
- GuardBlocks.pop_back();
-}
-
-/// We are using one integer to represent the block we are branching to. Then at
-/// each guard block, the predicate was calcuated using a simple `icmp eq`.
-static void calcPredicateUsingInteger(
- const BBSetVector &Incoming, const BBSetVector &Outgoing,
- SmallVectorImpl<BasicBlock *> &GuardBlocks, BBPredicates &GuardPredicates) {
- auto &Context = Incoming.front()->getContext();
- auto FirstGuardBlock = GuardBlocks.front();
-
- auto Phi = PHINode::Create(Type::getInt32Ty(Context), Incoming.size(),
- "merged.bb.idx", FirstGuardBlock);
-
- for (auto In : Incoming) {
- Value *Condition;
- BasicBlock *Succ0;
- BasicBlock *Succ1;
- std::tie(Condition, Succ0, Succ1) =
- redirectToHub(In, FirstGuardBlock, Outgoing);
- Value *IncomingId = nullptr;
- if (Succ0 && Succ1) {
- // target_bb_index = Condition ? index_of_succ0 : index_of_succ1.
- auto Succ0Iter = find(Outgoing, Succ0);
- auto Succ1Iter = find(Outgoing, Succ1);
- Value *Id0 = ConstantInt::get(Type::getInt32Ty(Context),
- std::distance(Outgoing.begin(), Succ0Iter));
- Value *Id1 = ConstantInt::get(Type::getInt32Ty(Context),
- std::distance(Outgoing.begin(), Succ1Iter));
- IncomingId = SelectInst::Create(Condition, Id0, Id1, "target.bb.idx",
- In->getTerminator()->getIterator());
- } else {
- // Get the index of the non-null successor.
- auto SuccIter = Succ0 ? find(Outgoing, Succ0) : find(Outgoing, Succ1);
- IncomingId = ConstantInt::get(Type::getInt32Ty(Context),
- std::distance(Outgoing.begin(), SuccIter));
- }
- Phi->addIncoming(IncomingId, In);
- }
-
- for (int i = 0, e = Outgoing.size() - 1; i != e; ++i) {
- auto Out = Outgoing[i];
- auto Cmp = ICmpInst::Create(Instruction::ICmp, ICmpInst::ICMP_EQ, Phi,
- ConstantInt::get(Type::getInt32Ty(Context), i),
- Out->getName() + ".predicate", GuardBlocks[i]);
- GuardPredicates[Out] = Cmp;
- }
-}
-
-/// We record the predicate of each outgoing block using a phi of boolean.
-static void calcPredicateUsingBooleans(
- const BBSetVector &Incoming, const BBSetVector &Outgoing,
- SmallVectorImpl<BasicBlock *> &GuardBlocks, BBPredicates &GuardPredicates,
- SmallVectorImpl<WeakVH> &DeletionCandidates) {
- auto &Context = Incoming.front()->getContext();
- auto BoolTrue = ConstantInt::getTrue(Context);
- auto BoolFalse = ConstantInt::getFalse(Context);
- auto FirstGuardBlock = GuardBlocks.front();
-
- // The predicate for the last outgoing is trivially true, and so we
- // process only the first N-1 successors.
- for (int i = 0, e = Outgoing.size() - 1; i != e; ++i) {
- auto Out = Outgoing[i];
- LLVM_DEBUG(dbgs() << "Creating guard for " << Out->getName() << "\n");
-
- auto Phi =
- PHINode::Create(Type::getInt1Ty(Context), Incoming.size(),
- StringRef("Guard.") + Out->getName(), FirstGuardBlock);
- GuardPredicates[Out] = Phi;
- }
-
- for (auto *In : Incoming) {
- Value *Condition;
- BasicBlock *Succ0;
- BasicBlock *Succ1;
- std::tie(Condition, Succ0, Succ1) =
- redirectToHub(In, FirstGuardBlock, Outgoing);
-
- // Optimization: Consider an incoming block A with both successors
- // Succ0 and Succ1 in the set of outgoing blocks. The predicates
- // for Succ0 and Succ1 complement each other. If Succ0 is visited
- // first in the loop below, control will branch to Succ0 using the
- // corresponding predicate. But if that branch is not taken, then
- // control must reach Succ1, which means that the incoming value of
- // the predicate from `In` is true for Succ1.
- bool OneSuccessorDone = false;
- for (int i = 0, e = Outgoing.size() - 1; i != e; ++i) {
- auto Out = Outgoing[i];
- PHINode *Phi = cast<PHINode>(GuardPredicates[Out]);
- if (Out != Succ0 && Out != Succ1) {
- Phi->addIncoming(BoolFalse, In);
- } else if (!Succ0 || !Succ1 || OneSuccessorDone) {
- // Optimization: When only one successor is an outgoing block,
- // the incoming predicate from `In` is always true.
- Phi->addIncoming(BoolTrue, In);
- } else {
- assert(Succ0 && Succ1);
- if (Out == Succ0) {
- Phi->addIncoming(Condition, In);
- } else {
- auto Inverted = invertCondition(Condition);
- DeletionCandidates.push_back(Condition);
- Phi->addIncoming(Inverted, In);
- }
- OneSuccessorDone = true;
- }
- }
- }
-}
-
-// Capture the existing control flow as guard predicates, and redirect
-// control flow from \p Incoming block through the \p GuardBlocks to the
-// \p Outgoing blocks.
-//
-// There is one guard predicate for each outgoing block OutBB. The
-// predicate represents whether the hub should transfer control flow
-// to OutBB. These predicates are NOT ORTHOGONAL. The Hub evaluates
-// them in the same order as the Outgoing set-vector, and control
-// branches to the first outgoing block whose predicate evaluates to true.
-static void
-convertToGuardPredicates(SmallVectorImpl<BasicBlock *> &GuardBlocks,
- SmallVectorImpl<WeakVH> &DeletionCandidates,
- const BBSetVector &Incoming,
- const BBSetVector &Outgoing, const StringRef Prefix,
- std::optional<unsigned> MaxControlFlowBooleans) {
- BBPredicates GuardPredicates;
- auto F = Incoming.front()->getParent();
-
- for (int i = 0, e = Outgoing.size() - 1; i != e; ++i)
- GuardBlocks.push_back(
- BasicBlock::Create(F->getContext(), Prefix + ".guard", F));
-
- // When we are using an integer to record which target block to jump to, we
- // are creating less live values, actually we are using one single integer to
- // store the index of the target block. When we are using booleans to store
- // the branching information, we need (N-1) boolean values, where N is the
- // number of outgoing block.
- if (!MaxControlFlowBooleans || Outgoing.size() <= *MaxControlFlowBooleans)
- calcPredicateUsingBooleans(Incoming, Outgoing, GuardBlocks, GuardPredicates,
- DeletionCandidates);
- else
- ...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/103013
More information about the llvm-branch-commits
mailing list