[llvm] [Transforms][Utils] Add LoopSplitUtils, a reusable iteration-space loop splitter (PR #209142)
Ashutosh Nema via llvm-commits
llvm-commits at lists.llvm.org
Mon Jul 13 04:27:27 PDT 2026
https://github.com/nema-ashutosh created https://github.com/llvm/llvm-project/pull/209142
This draft PR presents a patch stack that lays out the overall plan for a general-purpose loop-splitting utility and its adoption.
Its primary focus is introducing LoopSplitUtils as a reusable loop-splitting utility, and then demonstrating how existing passes such as IRCE and LoopBoundSplit can adopt it to perform their transformations.
The patch stack divided into following:
1. **[Transforms][Utils] Add LoopSplitUtils** - the core utility plus a
`loop-split-test` pass (driven from `opt`) and lit coverage. Initial scope:
unit-step, bottom-tested, single-exit loops with a computable trip count.
2. **Constant-step and top-tested loops** - generalize the induction analysis to
any non-zero constant step in either direction and to a counted exit in the
latch or the header.
3. **Loops with multi exits** - Extends support for multiple exit loops
4. **Uncomputable-trip-count and narrow-latch fallbacks** - two opt-in
relaxations (a symbolic counted bound with no exact trip count; a `trunc(iv)`
exit compare), both off by default.
5. **[IRCE]** - add `-irce-use-loop-split-utils` (hidden, default off) to route
IRCE's pre/main/post restructuring through the utility.
6. **[LoopBoundSplit]** - add `-loop-bound-split-use-loop-split-utils` (hidden,
default off) to route the bound split through the utility.
>From 569720c61bc23f738f320e80ddbf71aebaeb2c97 Mon Sep 17 00:00:00 2001
From: Ashutosh Nema <ashu1212 at gmail.com>
Date: Mon, 13 Jul 2026 16:21:59 +0530
Subject: [PATCH 1/6] Introduce LoopSplitUtils, a utility that splits a counted
loop into a chain of per-partition sub-loops covering contiguous slices of
the original iteration space. Given a loop and a list of partition ranges, it
clones the body per partition, guards each with an entry check that skips
empty partitions, clamps each latch to its slice, and rebuilds SSA for
loop-carried and live-out values so the result is behaviour-preserving.
Key properties:
Supports ascending (+1) and descending (-1) unit-step inductions, in
both signed and unsigned iteration orderings, with direction-aware
guard/latch predicates and end clamps.
Reuses the original loop for partition 0 and clones the rest, exposing
per-partition value maps via getPartitionValue()/getPartitionValueMap().
Lets callers drop the entry guard for a partition proven non-empty via
avoidPartitionGuard(); provably-empty partitions are always skipped.
Patches the dominator tree and LoopInfo incrementally rather than
rebuilding them.
How to use:
LoopSplitUtils LSU(L, LI, SE, DT);
if (!LSU.isLegal()) // counted, bottom-tested, LCSSA, unit step
return false;
// Tile the iteration space in order; e.g. split [Start, End] at K:
LSU.addPartition(Start, K - 1); // partition 0: [Start, K-1]
LSU.addPartition(K, End); // partition 1: [K, End]
LSU.split();
// After split(), query a cloned value in a given partition:
Value *V1 = LSU.getPartitionValue(Orig, /PartitionIndex=/1);
A new test pass, loop-split-test (opt -passes=loop-split-test with
-loop-split-points=...), drives the utility for testing. Adds lit tests
covering basic/multiple/four-partition splits, descending loops,
reductions, empty-leading partitions, optional guards, and the
per-partition value map.
---
.../llvm/Transforms/Utils/LoopSplitTestPass.h | 29 +
.../llvm/Transforms/Utils/LoopSplitUtils.h | 163 +++++
llvm/lib/Passes/PassBuilder.cpp | 1 +
llvm/lib/Passes/PassRegistry.def | 1 +
llvm/lib/Transforms/Utils/CMakeLists.txt | 2 +
.../Transforms/Utils/LoopSplitTestPass.cpp | 151 +++++
llvm/lib/Transforms/Utils/LoopSplitUtils.cpp | 595 ++++++++++++++++++
llvm/test/Transforms/LoopSplit/basic.ll | 62 ++
llvm/test/Transforms/LoopSplit/descending.ll | 61 ++
.../LoopSplit/empty-leading-partition.ll | 72 +++
.../Transforms/LoopSplit/four-partitions.ll | 89 +++
.../LoopSplit/multiple-partitions.ll | 76 +++
.../Transforms/LoopSplit/optional-guard.ll | 45 ++
.../LoopSplit/partition-value-map.ll | 44 ++
llvm/test/Transforms/LoopSplit/reduction.ll | 72 +++
15 files changed, 1463 insertions(+)
create mode 100644 llvm/include/llvm/Transforms/Utils/LoopSplitTestPass.h
create mode 100644 llvm/include/llvm/Transforms/Utils/LoopSplitUtils.h
create mode 100644 llvm/lib/Transforms/Utils/LoopSplitTestPass.cpp
create mode 100644 llvm/lib/Transforms/Utils/LoopSplitUtils.cpp
create mode 100644 llvm/test/Transforms/LoopSplit/basic.ll
create mode 100644 llvm/test/Transforms/LoopSplit/descending.ll
create mode 100644 llvm/test/Transforms/LoopSplit/empty-leading-partition.ll
create mode 100644 llvm/test/Transforms/LoopSplit/four-partitions.ll
create mode 100644 llvm/test/Transforms/LoopSplit/multiple-partitions.ll
create mode 100644 llvm/test/Transforms/LoopSplit/optional-guard.ll
create mode 100644 llvm/test/Transforms/LoopSplit/partition-value-map.ll
create mode 100644 llvm/test/Transforms/LoopSplit/reduction.ll
diff --git a/llvm/include/llvm/Transforms/Utils/LoopSplitTestPass.h b/llvm/include/llvm/Transforms/Utils/LoopSplitTestPass.h
new file mode 100644
index 0000000000000..1e427f02a542f
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Utils/LoopSplitTestPass.h
@@ -0,0 +1,29 @@
+//===- LoopSplitTestPass.h - Test driver for LoopSplitUtils -----*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// A command-line driven pass used to exercise the LoopSplitUtils utility from
+// `opt`. The split points are provided via the -loop-split-points option as
+// iteration offsets relative to the induction start.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TRANSFORMS_UTILS_LOOPSPLITTESTPASS_H
+#define LLVM_TRANSFORMS_UTILS_LOOPSPLITTESTPASS_H
+
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+class LoopSplitTestPass : public PassInfoMixin<LoopSplitTestPass> {
+public:
+ PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
+};
+
+} // namespace llvm
+
+#endif // LLVM_TRANSFORMS_UTILS_LOOPSPLITTESTPASS_H
diff --git a/llvm/include/llvm/Transforms/Utils/LoopSplitUtils.h b/llvm/include/llvm/Transforms/Utils/LoopSplitUtils.h
new file mode 100644
index 0000000000000..1d68e8db5d774
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Utils/LoopSplitUtils.h
@@ -0,0 +1,163 @@
+//===- LoopSplitUtils.h - Split a loop's iteration space --------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Splits a counted loop's iteration space into a chain of per-partition
+// sub-loops. See LoopSplitUtils.cpp for the structure produced.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TRANSFORMS_UTILS_LOOPSPLITUTILS_H
+#define LLVM_TRANSFORMS_UTILS_LOOPSPLITUTILS_H
+
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/Support/Compiler.h"
+#include "llvm/Transforms/Utils/ValueMapper.h"
+#include <memory>
+
+namespace llvm {
+
+class BasicBlock;
+class DominatorTree;
+class ICmpInst;
+class Instruction;
+class Loop;
+class LoopInfo;
+class PHINode;
+class SCEV;
+class SCEVAddRecExpr;
+class ScalarEvolution;
+class Value;
+
+/// Splits a counted loop into a chain of per-partition sub-loops.
+///
+/// Usage:
+/// \code
+/// LoopSplitUtils LSU(L, LI, SE, DT);
+/// if (!LSU.isLegal())
+/// return false;
+/// LSU.addPartition(S0, E0); // one call per partition, in order
+/// LSU.addPartition(S1, E1);
+/// LSU.split();
+/// \endcode
+class LoopSplitUtils {
+public:
+ LoopSplitUtils(Loop *L, LoopInfo *LI, ScalarEvolution *SE, DominatorTree *DT)
+ : L(L), LI(LI), SE(SE), DT(DT) {}
+
+ /// Analyze \p L and return true if it is a counted loop this utility can split:
+ /// a bottom-tested single-exit loop in LCSSA form with a unique unit-step
+ /// integer induction and a computable trip count. Must succeed before split().
+ LLVM_ABI bool isLegal();
+
+ /// Return the loop's induction variable. Valid only after isLegal() succeeds.
+ PHINode *getInductionVariable() const { return Induction; }
+
+ /// Append an inclusive partition range [Start, End] in iteration order.
+ /// Partitions must tile the whole space: first Start = induction start, each
+ /// later Start = previous End +/- step, last End = induction end (desc: S >= E).
+ ///
+ /// Bounds must be loop-invariant and representable in the induction type
+ /// without wrapping: a Start +/- offset that wraps past TYPE_MAX/MIN/0 looks
+ /// in-range and silently miscompiles. See LoopSplitUtils.cpp for the rationale.
+ ///
+ /// Every partition is guarded by default; use avoidPartitionGuard() to opt out.
+ LLVM_ABI void addPartition(const SCEV *Start, const SCEV *End);
+
+ /// Suppress the entry guard for partition \p PartitionIndex (already added). Use
+ /// only for a partition the caller can prove runs at least once; for a runtime-
+ /// empty partition this is incorrect and yields one spurious iteration.
+ LLVM_ABI void avoidPartitionGuard(unsigned PartitionIndex);
+
+ unsigned getNumPartitions() const { return Partitions.size(); }
+
+ /// Perform the split. Requires a successful isLegal() and at least two
+ /// partitions. Returns true if the loop was rewritten.
+ LLVM_ABI bool split();
+
+ /// Return the counterpart of original-loop value \p V in partition
+ /// \p PartitionIndex (0-based). Partition 0 maps values to themselves; a later
+ /// partition returns the clone, or null if not cloned. Valid only after split().
+ LLVM_ABI Value *getPartitionValue(const Value *V,
+ unsigned PartitionIndex) const;
+
+ /// Return the original-to-clone value map for the partition at
+ /// \p PartitionIndex, for callers that want to remap many values. Null for
+ /// partition 0 (identity) and for any partition that was not cloned.
+ LLVM_ABI const ValueToValueMapTy *
+ getPartitionValueMap(unsigned PartitionIndex) const;
+
+private:
+ /// Everything known about one partition: the caller-supplied range plus the
+ /// state split() derives. Indexed by partition number in \c Partitions.
+ struct PartitionInfo {
+ // Set by addPartition() / avoidPartitionGuard() before split():
+ const SCEV *StartExpr = nullptr; // inclusive iteration range [Start, End].
+ const SCEV *EndExpr = nullptr;
+ bool Guarded = true; // emit an entry guard?
+
+ // Filled in by split():
+ std::unique_ptr<ValueToValueMapTy> VMap; // null for partition 0 (identity).
+ Value *StartVal = nullptr; // expanded start.
+ Value *SelEnd = nullptr; // clamped end min(End, indEnd).
+ bool Empty = false; // provably zero-iteration.
+ BasicBlock *GuardBlock = nullptr;
+ BasicBlock *Preheader = nullptr;
+ BasicBlock *Exit = nullptr;
+ Loop *SubLoop = nullptr;
+ Value *LatchIndOp = nullptr; // induction operand of the latch compare.
+ };
+
+ /// Per-split() scratch threaded through the phase helpers (the escaping
+ /// values, new blocks, etc.). A pure transform internal, so it is defined in
+ /// the implementation file.
+ struct SplitState;
+
+ Loop *L;
+ LoopInfo *LI;
+ ScalarEvolution *SE;
+ DominatorTree *DT;
+
+ // Induction analysis, populated by isLegal().
+ PHINode *Induction = nullptr;
+ ICmpInst *LatchCmp = nullptr; // the loop's latch exit compare.
+ Value *LatchIndOperand = nullptr; // induction operand of the latch compare.
+ bool LatchUsesInductionPHI = false; // latch compares the PHI, not the step.
+ bool InductionIsSigned = false; // iteration ordering signedness.
+ bool InductionIsDescending = false; // step is -1 (loop counts down).
+ const SCEV *InductionEnd = nullptr;
+
+ /// One record per partition, in add order.
+ SmallVector<PartitionInfo, 4> Partitions;
+
+ /// Find and validate the induction recurrence; returns its add-recurrence, or
+ /// null if the loop has no suitable induction.
+ const SCEVAddRecExpr *analyzeInduction();
+ /// Determine the signedness of the iteration ordering from the latch compare
+ /// and the recurrence's no-wrap flags; returns false if it cannot be proven.
+ bool computeSignedness(const SCEVAddRecExpr *IndAR);
+
+ // split() phase helpers, run in order; each is documented at its definition.
+ /// Collect loop-carried and live-out values and split off the final exit.
+ void collectEscapingValues(SplitState &S);
+ /// Insert the entry guard ahead of partition 0 and update the dominator tree.
+ void buildEntryGuard(SplitState &S);
+ /// Expand each partition's start and clamped end into the entry guard.
+ void expandPartitionBounds(SplitState &S);
+ /// Pass 1: clone each later partition's sub-loop and create its guard/exit.
+ void clonePartitions(SplitState &S);
+ /// Pass 2: emit each guard, clamp each latch, and chain the partitions.
+ void chainPartitions(SplitState &S);
+ /// Rebuild SSA for every escaping value with a per-value SSAUpdater.
+ void reconstructSSA(SplitState &S);
+ /// Clamp \p PL's latch so it iterates only within [start, \p SelEnd].
+ void rewriteLatch(Loop *PL, Value *IndOp, Value *SelEnd, BasicBlock *Exit);
+};
+
+} // namespace llvm
+
+#endif // LLVM_TRANSFORMS_UTILS_LOOPSPLITUTILS_H
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index 603d7f2f5dea2..4674e4c3c5bd3 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -368,6 +368,7 @@
#include "llvm/Transforms/Utils/InstructionNamer.h"
#include "llvm/Transforms/Utils/LibCallsShrinkWrap.h"
#include "llvm/Transforms/Utils/LoopSimplify.h"
+#include "llvm/Transforms/Utils/LoopSplitTestPass.h"
#include "llvm/Transforms/Utils/LoopVersioning.h"
#include "llvm/Transforms/Utils/LowerGlobalDtors.h"
#include "llvm/Transforms/Utils/LowerIFunc.h"
diff --git a/llvm/lib/Passes/PassRegistry.def b/llvm/lib/Passes/PassRegistry.def
index 9edb30fedd867..7970434b91dfe 100644
--- a/llvm/lib/Passes/PassRegistry.def
+++ b/llvm/lib/Passes/PassRegistry.def
@@ -481,6 +481,7 @@ FUNCTION_PASS("loop-fusion", LoopFusePass())
FUNCTION_PASS("loop-load-elim", LoopLoadEliminationPass())
FUNCTION_PASS("loop-simplify", LoopSimplifyPass())
FUNCTION_PASS("loop-sink", LoopSinkPass())
+FUNCTION_PASS("loop-split-test", LoopSplitTestPass())
FUNCTION_PASS("loop-versioning", LoopVersioningPass())
FUNCTION_PASS("lower-atomic", LowerAtomicPass())
FUNCTION_PASS("lower-constant-intrinsics", LowerConstantIntrinsicsPass())
diff --git a/llvm/lib/Transforms/Utils/CMakeLists.txt b/llvm/lib/Transforms/Utils/CMakeLists.txt
index 933e204081ad2..6163a3019e487 100644
--- a/llvm/lib/Transforms/Utils/CMakeLists.txt
+++ b/llvm/lib/Transforms/Utils/CMakeLists.txt
@@ -48,6 +48,8 @@ add_llvm_component_library(LLVMTransformUtils
LoopPeel.cpp
LoopRotationUtils.cpp
LoopSimplify.cpp
+ LoopSplitTestPass.cpp
+ LoopSplitUtils.cpp
LoopUnroll.cpp
LoopUnrollAndJam.cpp
LoopUnrollRuntime.cpp
diff --git a/llvm/lib/Transforms/Utils/LoopSplitTestPass.cpp b/llvm/lib/Transforms/Utils/LoopSplitTestPass.cpp
new file mode 100644
index 0000000000000..b94ab1ee82496
--- /dev/null
+++ b/llvm/lib/Transforms/Utils/LoopSplitTestPass.cpp
@@ -0,0 +1,151 @@
+//===- LoopSplitTestPass.cpp - Test driver for LoopSplitUtils -------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass drives LoopSplitUtils from `opt` for testing. For every eligible
+// loop it builds partitions from the -loop-split-points offsets and splits the
+// loop.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Transforms/Utils/LoopSplitTestPass.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/Analysis/LoopInfo.h"
+#include "llvm/Analysis/ScalarEvolution.h"
+#include "llvm/Analysis/ScalarEvolutionExpressions.h"
+#include "llvm/IR/Dominators.h"
+#include "llvm/IR/Function.h"
+#include "llvm/IR/ValueHandle.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Support/raw_ostream.h"
+#include "llvm/Transforms/Utils/LoopSplitUtils.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "loop-split-test"
+
+static cl::list<unsigned>
+ SplitPoints("loop-split-points",
+ cl::desc("Iteration offsets (relative to the induction start) "
+ "at which to split each loop"),
+ cl::CommaSeparated);
+
+static cl::list<unsigned> UnguardedPartitions(
+ "loop-split-unguarded",
+ cl::desc("Partition indices whose entry guard is omitted (the caller "
+ "guarantees they run at least one iteration)"),
+ cl::CommaSeparated);
+
+static cl::opt<bool> PrintPartitionMap(
+ "loop-split-print-partition-map",
+ cl::desc("After splitting, print each original loop instruction's "
+ "counterpart in every partition (LoopSplitUtils::getPartitionValue)"),
+ cl::init(false));
+
+/// Build the partition list for \p L from the command-line split offsets and
+/// run the transform. Returns true if the loop was split.
+static bool splitLoop(Loop *L, ScalarEvolution &SE, DominatorTree &DT,
+ LoopInfo &LI) {
+ LoopSplitUtils LSU(L, &LI, &SE, &DT);
+ if (!LSU.isLegal()) {
+ LLVM_DEBUG(dbgs() << "loop-split-test: loop is not legal for splitting\n");
+ return false;
+ }
+
+ const auto *IndAR =
+ dyn_cast<SCEVAddRecExpr>(SE.getSCEV(LSU.getInductionVariable()));
+ if (!IndAR)
+ return false;
+
+ const SCEV *Start = IndAR->getStart();
+ const SCEV *BTC = SE.getBackedgeTakenCount(L);
+ const SCEV *End = IndAR->evaluateAtIteration(BTC, SE);
+ Type *Ty = Start->getType();
+ if (End->getType() != Ty)
+ End = SE.getTruncateExpr(End, Ty);
+
+ // Build boundaries in iteration order, stepping away from Start by each
+ // offset (down for a descending loop). Each offset opens a new partition at
+ // iteration `Start +/- offset`; the previous partition ends one step before.
+ bool Descending = false;
+ if (const auto *StepC = dyn_cast<SCEVConstant>(IndAR->getStepRecurrence(SE)))
+ Descending = StepC->getValue()->isMinusOne();
+
+ const SCEV *PrevStart = Start;
+ const SCEV *One = SE.getOne(Ty);
+ for (unsigned Offset : SplitPoints) {
+ const SCEV *Off = SE.getConstant(Ty, Offset);
+ const SCEV *Point =
+ Descending ? SE.getMinusSCEV(Start, Off) : SE.getAddExpr(Start, Off);
+ const SCEV *PrevEnd =
+ Descending ? SE.getAddExpr(Point, One) : SE.getMinusSCEV(Point, One);
+ LSU.addPartition(PrevStart, PrevEnd);
+ PrevStart = Point;
+ }
+ // The final partition runs to the iteration-space end.
+ LSU.addPartition(PrevStart, End);
+
+ // Suppress guards for the partitions the caller listed (out-of-range indices
+ // are ignored).
+ for (unsigned Idx : UnguardedPartitions)
+ if (Idx < LSU.getNumPartitions())
+ LSU.avoidPartitionGuard(Idx);
+
+ if (LSU.getNumPartitions() < 2)
+ return false;
+
+ // Snapshot the original loop's named instructions before the transform so we
+ // can query their per-partition counterparts afterwards (handles track any
+ // that the transform deletes).
+ SmallVector<WeakTrackingVH, 16> OrigValues;
+ if (PrintPartitionMap)
+ for (BasicBlock *BB : L->blocks())
+ for (Instruction &I : *BB)
+ if (I.hasName())
+ OrigValues.push_back(&I);
+
+ if (!LSU.split())
+ return false;
+
+ if (PrintPartitionMap) {
+ const unsigned N = LSU.getNumPartitions();
+ for (unsigned P = 0; P < N; ++P) {
+ outs() << "LS-MAP partition " << P << ":\n";
+ for (WeakTrackingVH &VH : OrigValues) {
+ if (!VH)
+ continue;
+ Value *M = LSU.getPartitionValue(VH, P);
+ outs() << "LS-MAP " << VH->getName() << " -> "
+ << (M ? M->getName() : "<none>") << "\n";
+ }
+ }
+ }
+ return true;
+}
+
+PreservedAnalyses LoopSplitTestPass::run(Function &F,
+ FunctionAnalysisManager &AM) {
+ if (SplitPoints.empty())
+ return PreservedAnalyses::all();
+
+ auto &LI = AM.getResult<LoopAnalysis>(F);
+ auto &SE = AM.getResult<ScalarEvolutionAnalysis>(F);
+ auto &DT = AM.getResult<DominatorTreeAnalysis>(F);
+
+ // Collect the original top-level loops up front; the transform creates new
+ // sub-loops that we must not revisit.
+ SmallVector<Loop *, 4> Worklist(LI.begin(), LI.end());
+
+ bool Changed = false;
+ for (Loop *L : Worklist) {
+ SE.forgetLoop(L);
+ Changed |= splitLoop(L, SE, DT, LI);
+ }
+
+ return Changed ? PreservedAnalyses::none() : PreservedAnalyses::all();
+}
diff --git a/llvm/lib/Transforms/Utils/LoopSplitUtils.cpp b/llvm/lib/Transforms/Utils/LoopSplitUtils.cpp
new file mode 100644
index 0000000000000..54a3d616f902f
--- /dev/null
+++ b/llvm/lib/Transforms/Utils/LoopSplitUtils.cpp
@@ -0,0 +1,595 @@
+//===- LoopSplitUtils.cpp - Split a loop's iteration space ----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Splits a counted loop's iteration space into a chain of per-partition
+// sub-loops. See LoopSplitUtils.h for the high-level usage guidelines.
+//
+// Structure produced for partitions [S0,E0], [S1,E1], ... where E is the loop's
+// last iteration and each clamped end sel_i = min(E_i, E):
+//
+// guard0: ; every S_i and sel_i is computed here
+// if (S0 <= sel0) goto preheader0 else goto guard1 ; default guard check
+// loop0: ... ; latch stops at sel0
+// exit0 -> guard1
+// guard1:
+// if (S1 <= sel1) goto preheader1 else goto guard2 ; default guard check
+// loop1: ... ; latch stops at sel1
+// exit1 -> guard2
+// ...
+// final.exit: ; merges every partition's live-outs
+//
+// Each guard holds the "S_i <= sel_i" check and skips an empty partition by
+// falling through to the next guard. The check is replaced by an unconditional
+// branch when a partition is proven empty (to the next guard) or the caller
+// exempts it via avoidPartitionGuard() (to its preheader). All S_i/sel_i are
+// materialized once in guard0; the end clamp keeps the "runs at least once"
+// iteration in the right partition; live-outs are rebuilt one SSAUpdater each.
+//
+// A descending (step -1) loop uses the same structure mirrored: partitions run
+// high-to-low and the empty test, clamp, and predicates flip (>=/>).
+//
+// Usage guidelines:
+// - Caller bounds must not wrap the induction type. The clamp absorbs a bound
+// past the runtime trip count, but a Start +/- offset that overshoots the
+// type extreme wraps in the bound arithmetic and cannot be repaired here.
+// - Bounds must be loop-invariant: they are expanded in guard0 (the preheader),
+// so a bound depending on a value defined inside the loop cannot be placed.
+// - The partitions must tile the original iteration space exactly -- same
+// iterations, same order -- so the split preserves program behaviour.
+// - A caller that drops a guard via avoidPartitionGuard() must itself ensure
+// that partition runs at least once, or the result is a spurious iteration.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Transforms/Utils/LoopSplitUtils.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/Analysis/LoopInfo.h"
+#include "llvm/Analysis/ScalarEvolution.h"
+#include "llvm/Analysis/ScalarEvolutionExpressions.h"
+#include "llvm/IR/BasicBlock.h"
+#include "llvm/IR/CFG.h"
+#include "llvm/IR/Constants.h"
+#include "llvm/IR/Dominators.h"
+#include "llvm/IR/Function.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/Instructions.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Transforms/Utils/BasicBlockUtils.h"
+#include "llvm/Transforms/Utils/Cloning.h"
+#include "llvm/Transforms/Utils/LoopUtils.h"
+#include "llvm/Transforms/Utils/SSAUpdater.h"
+#include "llvm/Transforms/Utils/ScalarEvolutionExpander.h"
+#include "llvm/Transforms/Utils/ValueMapper.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "loop-split-utils"
+
+//===----------------------------------------------------------------------===//
+// LoopSplitUtils - construction, partition list, induction analysis
+//===----------------------------------------------------------------------===//
+
+/// Per-split() scratch shared by the phase helpers; lives for one split() call.
+struct LoopSplitUtils::SplitState {
+ BasicBlock *OrigPreheader = nullptr; // also partition 0's preheader.
+ BasicBlock *ExitBlock = nullptr; // also partition 0's exit.
+ BasicBlock *FinalExit = nullptr; // where live-outs merge.
+ BasicBlock *EntryGuard = nullptr; // guard ahead of partition 0.
+ Loop *OuterLoop = nullptr; // parent of the new blocks, if any.
+
+ /// A value that must be reconstructed after cloning because it is
+ /// loop-carried (feeds a later partition), live-out (used after the loop), or
+ /// both.
+ struct EscapingValue {
+ /// The value as it exists in partition 0 (the original).
+ Value *Def = nullptr;
+ /// The carried header PHI in partition 0, or null if \c Def needs no
+ /// per-partition start value seeded.
+ PHINode *CarriedHeaderPHI = nullptr;
+ /// True if \c Def is used outside the loop and must be merged at the final
+ /// exit.
+ bool EscapesOutside = false;
+ /// \c Def and \c CarriedHeaderPHI cloned into each partition (index 0 is the
+ /// original; \c PerPartitionPHI[0] is unused).
+ SmallVector<Value *, 4> PerPartitionDef;
+ SmallVector<PHINode *, 4> PerPartitionPHI;
+ };
+
+ /// Values that must survive across partitions (carried and/or live-out).
+ SmallVector<EscapingValue, 8> Escaping;
+
+ EscapingValue &addEscaping(Value *Def) {
+ Escaping.emplace_back();
+ Escaping.back().Def = Def;
+ return Escaping.back();
+ }
+};
+
+// Record a new partition with the given inclusive iteration range.
+void LoopSplitUtils::addPartition(const SCEV *Start, const SCEV *End) {
+ PartitionInfo &P = Partitions.emplace_back();
+ P.StartExpr = Start;
+ P.EndExpr = End;
+}
+
+// Mark a partition so split() emits no entry guard for it.
+void LoopSplitUtils::avoidPartitionGuard(unsigned PartitionIndex) {
+ assert(PartitionIndex < Partitions.size() &&
+ "avoidPartitionGuard() called for an unknown partition");
+ Partitions[PartitionIndex].Guarded = false;
+}
+
+// Return a partition's original-to-clone map, or null if it has none.
+const ValueToValueMapTy *
+LoopSplitUtils::getPartitionValueMap(unsigned PartitionIndex) const {
+ if (PartitionIndex >= Partitions.size())
+ return nullptr;
+ return Partitions[PartitionIndex].VMap.get();
+}
+
+// Look up the counterpart of an original value in a given partition.
+Value *LoopSplitUtils::getPartitionValue(const Value *V,
+ unsigned PartitionIndex) const {
+ assert(PartitionIndex < getNumPartitions() && "partition index out of range");
+ // Partition 0 reuses the original loop: every value maps to itself.
+ if (PartitionIndex == 0)
+ return const_cast<Value *>(V);
+ const ValueToValueMapTy *VMap = getPartitionValueMap(PartitionIndex);
+ if (!VMap)
+ return nullptr;
+ ValueToValueMapTy::const_iterator It = VMap->find(V);
+ return It != VMap->end() ? It->second : nullptr;
+}
+
+// Find the induction variable and the latch operand it is compared against;
+// returns the induction's add-recurrence, or null if the loop is unsuitable.
+const SCEVAddRecExpr *LoopSplitUtils::analyzeInduction() {
+ // The loop must exit on an integer compare living in the latch.
+ LatchCmp = L->getLatchCmpInst();
+ if (!LatchCmp || LatchCmp->getParent() != L->getLoopLatch())
+ return nullptr;
+
+ // SCEV's induction variable, restricted to a unit-step affine recurrence.
+ Induction = L->getInductionVariable(*SE);
+ if (!Induction)
+ return nullptr;
+ const auto *AR = dyn_cast<SCEVAddRecExpr>(SE->getSCEV(Induction));
+ if (!AR || !AR->isAffine())
+ return nullptr;
+ // Accept a unit step in either direction: +1 (ascending) or -1 (descending).
+ const auto *Step = dyn_cast<SCEVConstant>(AR->getStepRecurrence(*SE));
+ if (!Step || !(Step->getValue()->isOne() || Step->getValue()->isMinusOne()))
+ return nullptr;
+ InductionIsDescending = Step->getValue()->isMinusOne();
+
+ // The induction's "next" value (i + 1), produced in the latch.
+ auto *StepInst = dyn_cast<Instruction>(
+ Induction->getIncomingValueForBlock(L->getLoopLatch()));
+ if (!StepInst)
+ return nullptr;
+
+ // Select the compare operand that is the induction (PHI or its step).
+ if (LatchCmp->getOperand(0) == Induction ||
+ LatchCmp->getOperand(0) == StepInst)
+ LatchIndOperand = LatchCmp->getOperand(0);
+ else if (LatchCmp->getOperand(1) == Induction ||
+ LatchCmp->getOperand(1) == StepInst)
+ LatchIndOperand = LatchCmp->getOperand(1);
+ else
+ return nullptr;
+ LatchUsesInductionPHI = (LatchIndOperand == Induction);
+ return AR;
+}
+
+// Decide whether the iteration ordering is signed or unsigned.
+bool LoopSplitUtils::computeSignedness(const SCEVAddRecExpr *IndAR) {
+ ICmpInst::Predicate P = LatchCmp->getPredicate();
+ // Relational predicate gives the ordering; for eq/ne use the no-wrap flags.
+ if (ICmpInst::isSigned(P))
+ InductionIsSigned = true;
+ else if (ICmpInst::isUnsigned(P))
+ InductionIsSigned = false;
+ else if (IndAR->hasNoSignedWrap())
+ InductionIsSigned = true;
+ else if (IndAR->hasNoUnsignedWrap())
+ InductionIsSigned = false;
+ else {
+ LLVM_DEBUG(dbgs() << "LS: cannot prove iteration ordering signedness\n");
+ return false;
+ }
+ return true;
+}
+
+// Check every structural precondition and record the induction analysis.
+bool LoopSplitUtils::isLegal() {
+ if (!L->getLoopPreheader() || !L->getLoopLatch()) {
+ LLVM_DEBUG(dbgs() << "LS: missing preheader/latch\n");
+ return false;
+ }
+ if (!L->getExitingBlock() || !L->getExitBlock() ||
+ L->getExitingBlock() != L->getLoopLatch()) {
+ LLVM_DEBUG(dbgs() << "LS: not a bottom-tested single-exit loop\n");
+ return false;
+ }
+ if (!L->isLCSSAForm(*DT)) {
+ LLVM_DEBUG(dbgs() << "LS: loop is not in LCSSA form\n");
+ return false;
+ }
+
+ // A computable backedge-taken count fixes the iteration space we rebuild.
+ const SCEV *BTC = SE->getBackedgeTakenCount(L);
+ if (isa<SCEVCouldNotCompute>(BTC)) {
+ LLVM_DEBUG(dbgs() << "LS: no computable trip count\n");
+ return false;
+ }
+
+ const SCEVAddRecExpr *IndAR = analyzeInduction();
+ if (!IndAR) {
+ LLVM_DEBUG(dbgs() << "LS: no unique unit-step integer induction\n");
+ return false;
+ }
+
+ if (!computeSignedness(IndAR))
+ return false;
+
+ InductionEnd = IndAR->evaluateAtIteration(BTC, *SE);
+ // Start and end must share the induction type; reject any width mismatch.
+ if (InductionEnd->getType() != IndAR->getStart()->getType()) {
+ LLVM_DEBUG(dbgs() << "LS: induction end/start type mismatch\n");
+ return false;
+ }
+ return true;
+}
+
+//===----------------------------------------------------------------------===//
+// Transform
+//===----------------------------------------------------------------------===//
+
+// Clone of \p V from \p VMap, or \p V itself if it was not cloned.
+static Value *remapValue(ValueToValueMapTy &VMap, Value *V) {
+ auto It = VMap.find(V);
+ if (It == VMap.end())
+ return V;
+ return It->second;
+}
+
+// Latch "keep iterating" predicate (ascending </<=, descending >/>=); inclusive
+// when the latch compares the step value, strict when it compares the PHI.
+static ICmpInst::Predicate continuePredicate(bool Signed, bool Descending,
+ bool Inclusive) {
+ if (Descending)
+ return Inclusive ? (Signed ? ICmpInst::ICMP_SGE : ICmpInst::ICMP_UGE)
+ : (Signed ? ICmpInst::ICMP_SGT : ICmpInst::ICMP_UGT);
+ return Inclusive ? (Signed ? ICmpInst::ICMP_SLE : ICmpInst::ICMP_ULE)
+ : (Signed ? ICmpInst::ICMP_SLT : ICmpInst::ICMP_ULT);
+}
+
+// Guard "enter this partition" predicate: Start <= sel ascending, Start >= sel
+// descending.
+static ICmpInst::Predicate guardPredicate(bool Signed, bool Descending) {
+ if (Descending)
+ return Signed ? ICmpInst::ICMP_SGE : ICmpInst::ICMP_UGE;
+ return Signed ? ICmpInst::ICMP_SLE : ICmpInst::ICMP_ULE;
+}
+
+// Drive the whole transform: set up scratch state and run each phase in order.
+bool LoopSplitUtils::split() {
+ assert(Induction && "split() requires a successful isLegal()");
+ if (getNumPartitions() < 2)
+ return false;
+
+ if (!L->hasDedicatedExits() &&
+ !formDedicatedExitBlocks(L, DT, LI, /*MSSAU=*/nullptr,
+ /*PreserveLCSSA=*/true))
+ return false;
+
+ SplitState S;
+ S.OrigPreheader = L->getLoopPreheader();
+ S.ExitBlock = L->getExitBlock();
+ S.OuterLoop = LI->getLoopFor(S.ExitBlock);
+
+ collectEscapingValues(S);
+ buildEntryGuard(S);
+ expandPartitionBounds(S);
+ clonePartitions(S);
+ chainPartitions(S);
+ reconstructSSA(S);
+ return true;
+}
+
+// Find loop-carried and live-out values and split the final-exit block off the
+// loop exit, seeding partition 0's slots for each escaping value.
+void LoopSplitUtils::collectEscapingValues(SplitState &S) {
+ BasicBlock *Latch = L->getLoopLatch();
+
+ // Separate FinalExit from the loop exit. Split at begin() so the LCSSA PHIs
+ // move into FinalExit (SplitBlock would advance past them).
+ S.FinalExit =
+ S.ExitBlock->splitBasicBlock(S.ExitBlock->begin(), "ls.final.exit");
+ if (S.OuterLoop)
+ S.OuterLoop->addBasicBlockToLoop(S.FinalExit, *LI);
+ // splitBasicBlock does not update the dominator tree; the new exit's sole
+ // predecessor is the original exit block.
+ DT->addNewBlock(S.FinalExit, S.ExitBlock);
+
+ // (1) Carried values: each non-induction header PHI whose backedge value
+ // differs from its initial value must resume in later partitions.
+ DenseMap<Value *, unsigned> CarriedDefToEscapingIdx;
+ for (PHINode &HeaderPHI : L->getHeader()->phis()) {
+ if (&HeaderPHI == Induction)
+ continue;
+ Value *CarriedValue = HeaderPHI.getIncomingValueForBlock(Latch);
+ Value *InitialValue = HeaderPHI.getIncomingValueForBlock(S.OrigPreheader);
+ if (CarriedValue == InitialValue)
+ continue; // invariant and equal to the initial value: nothing to carry.
+ auto &EV = S.addEscaping(CarriedValue);
+ EV.CarriedHeaderPHI = &HeaderPHI;
+ // Track in-loop carried defs so a matching live-out in (2) merges onto
+ // them.
+ if (auto *CarriedInst = dyn_cast<Instruction>(CarriedValue);
+ CarriedInst && L->contains(CarriedInst))
+ CarriedDefToEscapingIdx[CarriedValue] = S.Escaping.size() - 1;
+ }
+
+ // (2) Live-outs: dissolve each LCSSA PHI into its def and mark it escaping,
+ // merging onto a pass-(1) entry if also carried. Uses are repaired later.
+ for (PHINode &LCSSAPhi : make_early_inc_range(S.FinalExit->phis())) {
+ assert(LCSSAPhi.getNumIncomingValues() == 1 &&
+ "exit block not in LCSSA form");
+ Value *LiveOutDef = LCSSAPhi.getIncomingValue(0);
+ auto Existing = CarriedDefToEscapingIdx.find(LiveOutDef);
+ auto &EV = Existing != CarriedDefToEscapingIdx.end()
+ ? S.Escaping[Existing->second]
+ : S.addEscaping(LiveOutDef);
+ EV.EscapesOutside = true;
+ LCSSAPhi.replaceAllUsesWith(LiveOutDef);
+ LCSSAPhi.eraseFromParent();
+ }
+
+ // Seed partition 0 with the originals; later partitions are filled when
+ // cloned.
+ const unsigned N = getNumPartitions();
+ for (auto &EV : S.Escaping) {
+ EV.PerPartitionDef.assign(N, nullptr);
+ EV.PerPartitionPHI.assign(N, nullptr);
+ EV.PerPartitionDef[0] = EV.Def;
+ EV.PerPartitionPHI[0] = EV.CarriedHeaderPHI;
+ }
+}
+
+// Insert the entry guard ahead of partition 0's preheader and update the
+// dominator tree.
+void LoopSplitUtils::buildEntryGuard(SplitState &S) {
+ // Split the preheader: the upper half becomes the guard dominating the chain,
+ // the lower half a clean preheader.
+ std::string PreheaderName = S.OrigPreheader->getName().str();
+ BasicBlock *NewPreheader =
+ SplitBlock(S.OrigPreheader, S.OrigPreheader->getTerminator(), DT, LI);
+ S.EntryGuard = S.OrigPreheader;
+ S.OrigPreheader = NewPreheader;
+ S.EntryGuard->setName("ls.guard0");
+ S.OrigPreheader->setName(PreheaderName);
+}
+
+// Materialize each partition's start and clamped end in the entry guard and
+// flag the partitions that are provably empty at compile time.
+void LoopSplitUtils::expandPartitionBounds(SplitState &S) {
+ Type *IndTy = Induction->getType();
+ Instruction *EntryGuardTerm = S.EntryGuard->getTerminator();
+ SCEVExpander Expander(*SE, "ls");
+
+ // Expand all partition bounds in the entry guard, which dominates the whole
+ // chain (a skipped partition bypasses the original preheader).
+ const unsigned N = getNumPartitions();
+ for (unsigned I = 0; I < N; ++I) {
+ PartitionInfo &P = Partitions[I];
+
+ // Provably empty when Start overshoots End by exactly one step. Compile-time
+ // only: a runtime overshoot wraps at the type extreme and would falsely enter.
+ const SCEV *PartWidth = SE->getMinusSCEV(P.StartExpr, P.EndExpr);
+ if (auto *PartWidthConst = dyn_cast<SCEVConstant>(PartWidth)) {
+ const APInt &W = PartWidthConst->getAPInt();
+ P.Empty = InductionIsDescending ? W.isAllOnes() : W.isOne();
+ }
+
+ P.StartVal = Expander.expandCodeFor(P.StartExpr, IndTy, EntryGuardTerm);
+
+ // Clamp the end to the induction end (min ascending, max descending) so a
+ // short trip count keeps the last iteration in the right partition.
+ const SCEV *ClampedEndSCEV;
+ if (InductionIsDescending)
+ ClampedEndSCEV = InductionIsSigned ? SE->getSMaxExpr(P.EndExpr, InductionEnd)
+ : SE->getUMaxExpr(P.EndExpr, InductionEnd);
+ else
+ ClampedEndSCEV = InductionIsSigned ? SE->getSMinExpr(P.EndExpr, InductionEnd)
+ : SE->getUMinExpr(P.EndExpr, InductionEnd);
+ P.SelEnd = Expander.expandCodeFor(ClampedEndSCEV, IndTy, EntryGuardTerm);
+ }
+}
+
+// Pass 1: clone each later partition's sub-loop and create its guard and exit
+// blocks (partition 0 reuses the original loop).
+void LoopSplitUtils::clonePartitions(SplitState &S) {
+ Function &F = *L->getHeader()->getParent();
+ LLVMContext &Ctx = F.getContext();
+
+ const unsigned N = getNumPartitions();
+ // Partition 0 reuses the original blocks; its map stays null (identity).
+ PartitionInfo &P0 = Partitions[0];
+ P0.GuardBlock = S.EntryGuard;
+ P0.Preheader = S.OrigPreheader;
+ P0.Exit = S.ExitBlock;
+ P0.SubLoop = L;
+ P0.LatchIndOp = LatchIndOperand;
+
+ for (unsigned I = 1; I < N; ++I) {
+ PartitionInfo &P = Partitions[I];
+ // Persist this partition's original-to-clone map so callers can later
+ // query the counterpart of an original loop value (getPartitionValue()).
+ P.VMap = std::make_unique<ValueToValueMapTy>();
+ ValueToValueMapTy &VMap = *P.VMap;
+ SmallVector<BasicBlock *, 8> ClonedBlocks;
+ Loop *PL = cloneLoopWithPreheader(S.FinalExit, S.OrigPreheader, L, VMap,
+ ".ls" + Twine(I), LI, DT, ClonedBlocks);
+ remapInstructionsInBlocks(ClonedBlocks, VMap);
+ BasicBlock *PHi = PL->getLoopPreheader();
+
+ BasicBlock *Exiti =
+ BasicBlock::Create(Ctx, "ls.exit" + Twine(I), &F, S.FinalExit);
+ BasicBlock *Guardi =
+ BasicBlock::Create(Ctx, "ls.guard" + Twine(I), &F, PHi);
+ if (S.OuterLoop) {
+ S.OuterLoop->addBasicBlockToLoop(Exiti, *LI);
+ S.OuterLoop->addBasicBlockToLoop(Guardi, *LI);
+ }
+ // Placeholder terminators; both are re-pointed at the merge in pass 2.
+ IRBuilder<>(Exiti).CreateBr(S.FinalExit);
+ IRBuilder<>(Guardi).CreateBr(S.FinalExit);
+
+ // Seed the clone's induction PHI with this partition's start value.
+ auto *ClonedInduction = cast<PHINode>(VMap[Induction]);
+ ClonedInduction->setIncomingValueForBlock(PHi, P.StartVal);
+
+ P.GuardBlock = Guardi;
+ P.Preheader = PHi;
+ P.Exit = Exiti;
+ P.SubLoop = PL;
+ P.LatchIndOp = remapValue(VMap, LatchIndOperand);
+
+ for (auto &EV : S.Escaping) {
+ EV.PerPartitionDef[I] = remapValue(VMap, EV.Def);
+ if (EV.CarriedHeaderPHI)
+ EV.PerPartitionPHI[I] = cast<PHINode>(VMap[EV.CarriedHeaderPHI]);
+ }
+ }
+}
+
+// Replace a partition's latch test so it iterates only within [start, SelEnd].
+void LoopSplitUtils::rewriteLatch(Loop *PL, Value *IndOp, Value *SelEnd,
+ BasicBlock *Exit) {
+ auto *Term = cast<CondBrInst>(PL->getLoopLatch()->getTerminator());
+ auto *Cmp = cast<ICmpInst>(Term->getCondition());
+ IRBuilder<> B(Cmp);
+ Value *Bound = SelEnd;
+ if (Bound->getType() != IndOp->getType())
+ Bound = B.CreateIntCast(Bound, IndOp->getType(), InductionIsSigned);
+ // Strict when the PHI itself is compared, inclusive when the step value is.
+ ICmpInst::Predicate Pred = continuePredicate(
+ InductionIsSigned, InductionIsDescending, /*Inclusive=*/!LatchUsesInductionPHI);
+ Value *NewCmp = B.CreateICmp(Pred, IndOp, Bound, "itr.chk");
+ B.SetInsertPoint(Term);
+ B.CreateCondBr(NewCmp, PL->getHeader(), Exit);
+ Term->eraseFromParent();
+ if (Cmp->use_empty())
+ Cmp->eraseFromParent();
+}
+
+// Pass 2: emit each partition's guard branch, clamp its latch, wire the
+// partitions into a chain, and update the dominator tree.
+void LoopSplitUtils::chainPartitions(SplitState &S) {
+ const ICmpInst::Predicate GuardPred =
+ guardPredicate(InductionIsSigned, InductionIsDescending);
+
+ // Emit each guard, clamp each latch, and chain partitions; a skipped
+ // partition falls through to the next guard.
+ const unsigned N = getNumPartitions();
+
+ // Enters unconditionally when the caller opted out of the guard and the
+ // partition is not provably empty; a proven-empty partition always skips.
+ auto EntersUnconditionally = [](const PartitionInfo &P) {
+ return !P.Empty && !P.Guarded;
+ };
+
+ // Where control goes when partition Idx is skipped or after it finishes: the
+ // next partition's guard, or the final merge block for the last partition.
+ auto MergeTargetAfter = [&](unsigned Idx) -> BasicBlock * {
+ bool IsLastPartition = Idx + 1 == N;
+ return IsLastPartition ? S.FinalExit : Partitions[Idx + 1].GuardBlock;
+ };
+
+ for (unsigned I = 0; I < N; ++I) {
+ PartitionInfo &P = Partitions[I];
+ BasicBlock *MergeAfter = MergeTargetAfter(I);
+
+ Instruction *GuardTerm = P.GuardBlock->getTerminator();
+ if (P.Empty) {
+ // Provably empty: skip to the next partition. The unreachable loop body
+ // is removed by later passes.
+ IRBuilder<>(GuardTerm).CreateBr(MergeAfter);
+ } else if (!P.Guarded) {
+ // Caller guaranteed at least one iteration: enter unconditionally. The
+ // skip edge to MergeAfter is omitted (see DT update below).
+ IRBuilder<>(GuardTerm).CreateBr(P.Preheader);
+ } else {
+ IRBuilder<> B(GuardTerm);
+ Value *Enter = B.CreateICmp(GuardPred, P.StartVal, P.SelEnd, "itr.chk");
+ B.CreateCondBr(Enter, P.Preheader, MergeAfter);
+ }
+ GuardTerm->eraseFromParent();
+
+ rewriteLatch(P.SubLoop, P.LatchIndOp, P.SelEnd, P.Exit);
+ P.Exit->getTerminator()->setSuccessor(0, MergeAfter);
+ }
+
+ // Patch the dominator tree directly: a merge target is dominated by the prior
+ // partition's exit when it enters unconditionally, otherwise by its guard.
+ auto MergeTargetIDom = [&](const PartitionInfo &P) {
+ return EntersUnconditionally(P) ? P.Exit : P.GuardBlock;
+ };
+
+ for (unsigned I = 1; I < N; ++I) {
+ PartitionInfo &Prev = Partitions[I - 1];
+ PartitionInfo &Cur = Partitions[I];
+ DT->addNewBlock(Cur.GuardBlock, MergeTargetIDom(Prev));
+ DT->changeImmediateDominator(Cur.Preheader, Cur.GuardBlock);
+ DT->addNewBlock(Cur.Exit, Cur.SubLoop->getLoopLatch());
+ }
+ // The final exit is the last partition's merge target.
+ DT->changeImmediateDominator(S.FinalExit, MergeTargetIDom(Partitions.back()));
+}
+
+// Rebuild SSA for every escaping value, repairing outside uses and seeding each
+// later partition's carried PHI, using one SSAUpdater per value.
+void LoopSplitUtils::reconstructSSA(SplitState &S) {
+ const unsigned N = getNumPartitions();
+ for (auto &EV : S.Escaping) {
+ SSAUpdater Updater;
+ Updater.Initialize(EV.Def->getType(), EV.Def->getName());
+
+ // Value before any partition runs: carried PHI's initial value, else
+ // poison.
+ Value *Init =
+ EV.CarriedHeaderPHI
+ ? EV.CarriedHeaderPHI->getIncomingValueForBlock(S.OrigPreheader)
+ : PoisonValue::get(EV.Def->getType());
+ Updater.AddAvailableValue(S.EntryGuard, Init);
+ for (unsigned I = 0; I < N; ++I)
+ Updater.AddAvailableValue(Partitions[I].Exit, EV.PerPartitionDef[I]);
+
+ // Repair outside uses before the carried-PHI seeds add new in-clone uses.
+ if (EV.EscapesOutside) {
+ SmallVector<Use *, 8> OutsideUses;
+ for (Use &U : EV.Def->uses())
+ if (auto *User = dyn_cast<Instruction>(U.getUser()))
+ if (!L->contains(User))
+ OutsideUses.push_back(&U);
+ for (Use *U : OutsideUses)
+ Updater.RewriteUse(*U);
+ }
+
+ // Seed each later partition's carried PHI from the preceding partitions.
+ if (EV.CarriedHeaderPHI)
+ for (unsigned I = 1; I < N; ++I) {
+ PHINode *CarriedPHI = EV.PerPartitionPHI[I];
+ int PreheaderEntryIdx =
+ CarriedPHI->getBasicBlockIndex(Partitions[I].Preheader);
+ assert(PreheaderEntryIdx >= 0 && "cloned preheader edge missing");
+ Updater.RewriteUse(CarriedPHI->getOperandUse(PreheaderEntryIdx));
+ }
+ }
+}
diff --git a/llvm/test/Transforms/LoopSplit/basic.ll b/llvm/test/Transforms/LoopSplit/basic.ll
new file mode 100644
index 0000000000000..48e836bdb7cb0
--- /dev/null
+++ b/llvm/test/Transforms/LoopSplit/basic.ll
@@ -0,0 +1,62 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt -passes='loop-simplify,lcssa,loop-split-test,verify' \
+; RUN: -loop-split-points=50 -S < %s | FileCheck %s
+
+; A single counted loop split into two partitions at iteration 50.
+
+define void @basic(ptr %a, i64 %n) {
+; CHECK-LABEL: define void @basic(
+; CHECK-SAME: ptr [[A:%.*]], i64 [[N:%.*]]) {
+; CHECK-NEXT: [[LS_GUARD0:.*:]]
+; CHECK-NEXT: [[SMAX:%.*]] = call i64 @llvm.smax.i64(i64 [[N]], i64 1)
+; CHECK-NEXT: [[TMP0:%.*]] = add nsw i64 [[SMAX]], -1
+; CHECK-NEXT: [[SMIN:%.*]] = call i64 @llvm.smin.i64(i64 [[TMP0]], i64 49)
+; CHECK-NEXT: [[ITR_CHK:%.*]] = icmp sle i64 0, [[SMIN]]
+; CHECK-NEXT: br i1 [[ITR_CHK]], label %[[ENTRY:.*]], label %[[LS_GUARD1:.*]]
+; CHECK: [[ENTRY]]:
+; CHECK-NEXT: br label %[[LOOP:.*]]
+; CHECK: [[LOOP]]:
+; CHECK-NEXT: [[I:%.*]] = phi i64 [ 0, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[LOOP]] ]
+; CHECK-NEXT: [[P:%.*]] = getelementptr i64, ptr [[A]], i64 [[I]]
+; CHECK-NEXT: store i64 [[I]], ptr [[P]], align 4
+; CHECK-NEXT: [[I_NEXT]] = add i64 [[I]], 1
+; CHECK-NEXT: [[ITR_CHK1:%.*]] = icmp sle i64 [[I_NEXT]], [[SMIN]]
+; CHECK-NEXT: br i1 [[ITR_CHK1]], label %[[LOOP]], label %[[EXIT:.*]]
+; CHECK: [[EXIT]]:
+; CHECK-NEXT: br label %[[LS_GUARD1]]
+; CHECK: [[LS_GUARD1]]:
+; CHECK-NEXT: [[ITR_CHK2:%.*]] = icmp sle i64 50, [[TMP0]]
+; CHECK-NEXT: br i1 [[ITR_CHK2]], label %[[ENTRY_LS1:.*]], label %[[LS_FINAL_EXIT:.*]]
+; CHECK: [[ENTRY_LS1]]:
+; CHECK-NEXT: br label %[[LOOP_LS1:.*]]
+; CHECK: [[LOOP_LS1]]:
+; CHECK-NEXT: [[I_LS1:%.*]] = phi i64 [ 50, %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
+; CHECK-NEXT: [[P_LS1:%.*]] = getelementptr i64, ptr [[A]], i64 [[I_LS1]]
+; CHECK-NEXT: store i64 [[I_LS1]], ptr [[P_LS1]], align 4
+; CHECK-NEXT: [[I_NEXT_LS1]] = add i64 [[I_LS1]], 1
+; CHECK-NEXT: [[ITR_CHK3:%.*]] = icmp sle i64 [[I_NEXT_LS1]], [[TMP0]]
+; CHECK-NEXT: br i1 [[ITR_CHK3]], label %[[LOOP_LS1]], label %[[LS_EXIT1:.*]]
+; CHECK: [[LS_EXIT1]]:
+; CHECK-NEXT: br label %[[LS_FINAL_EXIT]]
+; CHECK: [[LS_FINAL_EXIT]]:
+; CHECK-NEXT: ret void
+;
+; Each partition's latch is clamped to min(partitionEnd, loopEnd), emitted as a
+; signed-min intrinsic for a signed induction.
+; A later partition is entered only if a runtime check passes, and a second,
+; cloned loop drives the remaining iterations.
+; The transform builds a dedicated final-exit block joining all partitions.
+entry:
+ br label %loop
+
+loop:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %loop ]
+ %p = getelementptr i64, ptr %a, i64 %i
+ store i64 %i, ptr %p
+ %i.next = add i64 %i, 1
+ %c = icmp slt i64 %i.next, %n
+ br i1 %c, label %loop, label %exit
+
+exit:
+ ret void
+}
diff --git a/llvm/test/Transforms/LoopSplit/descending.ll b/llvm/test/Transforms/LoopSplit/descending.ll
new file mode 100644
index 0000000000000..e6e1b56acd5f5
--- /dev/null
+++ b/llvm/test/Transforms/LoopSplit/descending.ll
@@ -0,0 +1,61 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt -passes='loop-simplify,lcssa,loop-split-test,verify' \
+; RUN: -loop-split-points=50 -S < %s | FileCheck %s
+
+; A signed counting-down loop (step -1) split into two partitions. The guard and
+; latch predicates flip to >=, and the end clamp uses smax instead of smin.
+
+define void @descending(ptr %a, i64 %n) {
+; CHECK-LABEL: define void @descending(
+; CHECK-SAME: ptr [[A:%.*]], i64 [[N:%.*]]) {
+; CHECK-NEXT: [[LS_GUARD0:.*:]]
+; CHECK-NEXT: [[TMP0:%.*]] = add i64 [[N]], -49
+; CHECK-NEXT: [[TMP1:%.*]] = add i64 [[N]], -1
+; CHECK-NEXT: [[SMIN:%.*]] = call i64 @llvm.smin.i64(i64 [[TMP1]], i64 -1)
+; CHECK-NEXT: [[TMP2:%.*]] = add nsw i64 [[SMIN]], 1
+; CHECK-NEXT: [[SMAX:%.*]] = call i64 @llvm.smax.i64(i64 [[TMP0]], i64 [[TMP2]])
+; CHECK-NEXT: [[TMP3:%.*]] = add i64 [[N]], -50
+; CHECK-NEXT: [[ITR_CHK:%.*]] = icmp sge i64 [[N]], [[SMAX]]
+; CHECK-NEXT: br i1 [[ITR_CHK]], label %[[ENTRY:.*]], label %[[LS_GUARD1:.*]]
+; CHECK: [[ENTRY]]:
+; CHECK-NEXT: br label %[[LOOP:.*]]
+; CHECK: [[LOOP]]:
+; CHECK-NEXT: [[I:%.*]] = phi i64 [ [[N]], %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[LOOP]] ]
+; CHECK-NEXT: [[P:%.*]] = getelementptr i64, ptr [[A]], i64 [[I]]
+; CHECK-NEXT: store i64 [[I]], ptr [[P]], align 4
+; CHECK-NEXT: [[I_NEXT]] = add i64 [[I]], -1
+; CHECK-NEXT: [[ITR_CHK1:%.*]] = icmp sge i64 [[I_NEXT]], [[SMAX]]
+; CHECK-NEXT: br i1 [[ITR_CHK1]], label %[[LOOP]], label %[[EXIT:.*]]
+; CHECK: [[EXIT]]:
+; CHECK-NEXT: br label %[[LS_GUARD1]]
+; CHECK: [[LS_GUARD1]]:
+; CHECK-NEXT: [[ITR_CHK2:%.*]] = icmp sge i64 [[TMP3]], [[TMP2]]
+; CHECK-NEXT: br i1 [[ITR_CHK2]], label %[[ENTRY_LS1:.*]], label %[[LS_FINAL_EXIT:.*]]
+; CHECK: [[ENTRY_LS1]]:
+; CHECK-NEXT: br label %[[LOOP_LS1:.*]]
+; CHECK: [[LOOP_LS1]]:
+; CHECK-NEXT: [[I_LS1:%.*]] = phi i64 [ [[TMP3]], %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
+; CHECK-NEXT: [[P_LS1:%.*]] = getelementptr i64, ptr [[A]], i64 [[I_LS1]]
+; CHECK-NEXT: store i64 [[I_LS1]], ptr [[P_LS1]], align 4
+; CHECK-NEXT: [[I_NEXT_LS1]] = add i64 [[I_LS1]], -1
+; CHECK-NEXT: [[ITR_CHK3:%.*]] = icmp sge i64 [[I_NEXT_LS1]], [[TMP2]]
+; CHECK-NEXT: br i1 [[ITR_CHK3]], label %[[LOOP_LS1]], label %[[LS_EXIT1:.*]]
+; CHECK: [[LS_EXIT1]]:
+; CHECK-NEXT: br label %[[LS_FINAL_EXIT]]
+; CHECK: [[LS_FINAL_EXIT]]:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %loop
+
+loop:
+ %i = phi i64 [ %n, %entry ], [ %i.next, %loop ]
+ %p = getelementptr i64, ptr %a, i64 %i
+ store i64 %i, ptr %p
+ %i.next = add i64 %i, -1
+ %c = icmp sge i64 %i.next, 0
+ br i1 %c, label %loop, label %exit
+
+exit:
+ ret void
+}
diff --git a/llvm/test/Transforms/LoopSplit/empty-leading-partition.ll b/llvm/test/Transforms/LoopSplit/empty-leading-partition.ll
new file mode 100644
index 0000000000000..dd0660a43c83d
--- /dev/null
+++ b/llvm/test/Transforms/LoopSplit/empty-leading-partition.ll
@@ -0,0 +1,72 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt -passes='loop-simplify,lcssa,loop-split-test,verify' \
+; RUN: -loop-split-points=0 -S < %s | FileCheck %s
+
+; A leading split point of 0 makes partition 0 empty ([Start, Start-1]). The
+; loop is bottom-tested, so without an entry guard partition 0 would still run
+; one iteration and the loop would execute its first iteration twice. The empty
+; partition is detected at compile time (its width S - E folds to a constant)
+; and skipped: partition 0's guard unconditionally falls through to partition 1,
+; which runs the whole iteration space and seeds from the chain-entry value (0).
+
+define i64 @reduction(ptr %a, i64 %n) {
+; CHECK-LABEL: define i64 @reduction(
+; CHECK-SAME: ptr [[A:%.*]], i64 [[N:%.*]]) {
+; CHECK-NEXT: [[LS_GUARD0:.*]]:
+; CHECK-NEXT: [[SMAX:%.*]] = call i64 @llvm.smax.i64(i64 [[N]], i64 1)
+; CHECK-NEXT: [[TMP0:%.*]] = add nsw i64 [[SMAX]], -1
+; CHECK-NEXT: br label %[[LS_GUARD1:.*]]
+; CHECK: [[ENTRY:.*]]:
+; CHECK-NEXT: br label %[[LOOP:.*]]
+; CHECK: [[LOOP]]:
+; CHECK-NEXT: [[I:%.*]] = phi i64 [ 0, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[LOOP]] ]
+; CHECK-NEXT: [[SUM:%.*]] = phi i64 [ 0, %[[ENTRY]] ], [ [[SUM_NEXT:%.*]], %[[LOOP]] ]
+; CHECK-NEXT: [[P:%.*]] = getelementptr i64, ptr [[A]], i64 [[I]]
+; CHECK-NEXT: [[V:%.*]] = load i64, ptr [[P]], align 4
+; CHECK-NEXT: [[SUM_NEXT]] = add i64 [[SUM]], [[V]]
+; CHECK-NEXT: [[I_NEXT]] = add i64 [[I]], 1
+; CHECK-NEXT: [[ITR_CHK:%.*]] = icmp sle i64 [[I_NEXT]], -1
+; CHECK-NEXT: br i1 [[ITR_CHK]], label %[[LOOP]], label %[[EXIT:.*]]
+; CHECK: [[EXIT]]:
+; CHECK-NEXT: br label %[[LS_GUARD1]]
+; CHECK: [[LS_GUARD1]]:
+; CHECK-NEXT: [[SUM_NEXT4:%.*]] = phi i64 [ [[SUM_NEXT]], %[[EXIT]] ], [ 0, %[[LS_GUARD0]] ]
+; CHECK-NEXT: [[ITR_CHK1:%.*]] = icmp sle i64 0, [[TMP0]]
+; CHECK-NEXT: br i1 [[ITR_CHK1]], label %[[ENTRY_LS1:.*]], label %[[LS_FINAL_EXIT:.*]]
+; CHECK: [[ENTRY_LS1]]:
+; CHECK-NEXT: br label %[[LOOP_LS1:.*]]
+; CHECK: [[LOOP_LS1]]:
+; CHECK-NEXT: [[I_LS1:%.*]] = phi i64 [ 0, %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
+; CHECK-NEXT: [[SUM_LS1:%.*]] = phi i64 [ [[SUM_NEXT4]], %[[ENTRY_LS1]] ], [ [[SUM_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
+; CHECK-NEXT: [[P_LS1:%.*]] = getelementptr i64, ptr [[A]], i64 [[I_LS1]]
+; CHECK-NEXT: [[V_LS1:%.*]] = load i64, ptr [[P_LS1]], align 4
+; CHECK-NEXT: [[SUM_NEXT_LS1]] = add i64 [[SUM_LS1]], [[V_LS1]]
+; CHECK-NEXT: [[I_NEXT_LS1]] = add i64 [[I_LS1]], 1
+; CHECK-NEXT: [[ITR_CHK2:%.*]] = icmp sle i64 [[I_NEXT_LS1]], [[TMP0]]
+; CHECK-NEXT: br i1 [[ITR_CHK2]], label %[[LOOP_LS1]], label %[[LS_EXIT1:.*]]
+; CHECK: [[LS_EXIT1]]:
+; CHECK-NEXT: br label %[[LS_FINAL_EXIT]]
+; CHECK: [[LS_FINAL_EXIT]]:
+; CHECK-NEXT: [[SUM_NEXT3:%.*]] = phi i64 [ [[SUM_NEXT_LS1]], %[[LS_EXIT1]] ], [ [[SUM_NEXT4]], %[[LS_GUARD1]] ]
+; CHECK-NEXT: ret i64 [[SUM_NEXT3]]
+;
+; The empty leading partition is never entered: an unconditional fall-through.
+; The original loop body is now dead (unreachable), proving it is not executed.
+; The surviving partition runs the full range, seeded from the chain-entry value.
+entry:
+ br label %loop
+
+loop:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %loop ]
+ %sum = phi i64 [ 0, %entry ], [ %sum.next, %loop ]
+ %p = getelementptr i64, ptr %a, i64 %i
+ %v = load i64, ptr %p
+ %sum.next = add i64 %sum, %v
+ %i.next = add i64 %i, 1
+ %c = icmp slt i64 %i.next, %n
+ br i1 %c, label %loop, label %exit
+
+exit:
+ %sum.lcssa = phi i64 [ %sum.next, %loop ]
+ ret i64 %sum.lcssa
+}
diff --git a/llvm/test/Transforms/LoopSplit/four-partitions.ll b/llvm/test/Transforms/LoopSplit/four-partitions.ll
new file mode 100644
index 0000000000000..4ebd030691f2d
--- /dev/null
+++ b/llvm/test/Transforms/LoopSplit/four-partitions.ll
@@ -0,0 +1,89 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt -passes='loop-simplify,lcssa,loop-split-test,verify' \
+; RUN: -loop-split-points=100,200,300 -S < %s | FileCheck %s
+
+; A minimal counted loop split into four partitions: [0,99], [100,199],
+; [200,299] and [300,n-1]. Three clones (loop.ls1, loop.ls2, loop.ls3) chain
+; through their guards into a single dedicated final-exit block.
+
+define void @four(ptr %a, i64 %n) {
+; CHECK-LABEL: define void @four(
+; CHECK-SAME: ptr [[A:%.*]], i64 [[N:%.*]]) {
+; CHECK-NEXT: [[LS_GUARD0:.*:]]
+; CHECK-NEXT: [[SMAX:%.*]] = call i64 @llvm.smax.i64(i64 [[N]], i64 1)
+; CHECK-NEXT: [[TMP0:%.*]] = add nsw i64 [[SMAX]], -1
+; CHECK-NEXT: [[SMIN2:%.*]] = call i64 @llvm.smin.i64(i64 [[TMP0]], i64 99)
+; CHECK-NEXT: [[SMIN:%.*]] = call i64 @llvm.smin.i64(i64 [[TMP0]], i64 199)
+; CHECK-NEXT: [[SMIN1:%.*]] = call i64 @llvm.smin.i64(i64 [[TMP0]], i64 299)
+; CHECK-NEXT: [[ITR_CHK1:%.*]] = icmp sle i64 0, [[SMIN2]]
+; CHECK-NEXT: br i1 [[ITR_CHK1]], label %[[ENTRY:.*]], label %[[LS_GUARD1:.*]]
+; CHECK: [[ENTRY]]:
+; CHECK-NEXT: br label %[[LOOP:.*]]
+; CHECK: [[LOOP]]:
+; CHECK-NEXT: [[I:%.*]] = phi i64 [ 0, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[LOOP]] ]
+; CHECK-NEXT: [[P:%.*]] = getelementptr i64, ptr [[A]], i64 [[I]]
+; CHECK-NEXT: store i64 [[I]], ptr [[P]], align 4
+; CHECK-NEXT: [[I_NEXT]] = add i64 [[I]], 1
+; CHECK-NEXT: [[ITR_CHK:%.*]] = icmp sle i64 [[I_NEXT]], [[SMIN2]]
+; CHECK-NEXT: br i1 [[ITR_CHK]], label %[[LOOP]], label %[[EXIT:.*]]
+; CHECK: [[EXIT]]:
+; CHECK-NEXT: br label %[[LS_GUARD1]]
+; CHECK: [[LS_GUARD1]]:
+; CHECK-NEXT: [[ITR_CHK2:%.*]] = icmp sle i64 100, [[SMIN]]
+; CHECK-NEXT: br i1 [[ITR_CHK2]], label %[[ENTRY_LS1:.*]], label %[[LS_GUARD2:.*]]
+; CHECK: [[ENTRY_LS1]]:
+; CHECK-NEXT: br label %[[LOOP_LS1:.*]]
+; CHECK: [[LOOP_LS1]]:
+; CHECK-NEXT: [[I_LS1:%.*]] = phi i64 [ 100, %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
+; CHECK-NEXT: [[P_LS1:%.*]] = getelementptr i64, ptr [[A]], i64 [[I_LS1]]
+; CHECK-NEXT: store i64 [[I_LS1]], ptr [[P_LS1]], align 4
+; CHECK-NEXT: [[I_NEXT_LS1]] = add i64 [[I_LS1]], 1
+; CHECK-NEXT: [[ITR_CHK3:%.*]] = icmp sle i64 [[I_NEXT_LS1]], [[SMIN]]
+; CHECK-NEXT: br i1 [[ITR_CHK3]], label %[[LOOP_LS1]], label %[[LS_EXIT1:.*]]
+; CHECK: [[LS_EXIT1]]:
+; CHECK-NEXT: br label %[[LS_GUARD2]]
+; CHECK: [[LS_GUARD2]]:
+; CHECK-NEXT: [[ITR_CHK4:%.*]] = icmp sle i64 200, [[SMIN1]]
+; CHECK-NEXT: br i1 [[ITR_CHK4]], label %[[ENTRY_LS2:.*]], label %[[LS_GUARD3:.*]]
+; CHECK: [[ENTRY_LS2]]:
+; CHECK-NEXT: br label %[[LOOP_LS2:.*]]
+; CHECK: [[LOOP_LS2]]:
+; CHECK-NEXT: [[I_LS2:%.*]] = phi i64 [ 200, %[[ENTRY_LS2]] ], [ [[I_NEXT_LS2:%.*]], %[[LOOP_LS2]] ]
+; CHECK-NEXT: [[P_LS2:%.*]] = getelementptr i64, ptr [[A]], i64 [[I_LS2]]
+; CHECK-NEXT: store i64 [[I_LS2]], ptr [[P_LS2]], align 4
+; CHECK-NEXT: [[I_NEXT_LS2]] = add i64 [[I_LS2]], 1
+; CHECK-NEXT: [[ITR_CHK5:%.*]] = icmp sle i64 [[I_NEXT_LS2]], [[SMIN1]]
+; CHECK-NEXT: br i1 [[ITR_CHK5]], label %[[LOOP_LS2]], label %[[LS_EXIT2:.*]]
+; CHECK: [[LS_EXIT2]]:
+; CHECK-NEXT: br label %[[LS_GUARD3]]
+; CHECK: [[LS_GUARD3]]:
+; CHECK-NEXT: [[ITR_CHK6:%.*]] = icmp sle i64 300, [[TMP0]]
+; CHECK-NEXT: br i1 [[ITR_CHK6]], label %[[ENTRY_LS3:.*]], label %[[LS_FINAL_EXIT:.*]]
+; CHECK: [[ENTRY_LS3]]:
+; CHECK-NEXT: br label %[[LOOP_LS3:.*]]
+; CHECK: [[LOOP_LS3]]:
+; CHECK-NEXT: [[I_LS3:%.*]] = phi i64 [ 300, %[[ENTRY_LS3]] ], [ [[I_NEXT_LS3:%.*]], %[[LOOP_LS3]] ]
+; CHECK-NEXT: [[P_LS3:%.*]] = getelementptr i64, ptr [[A]], i64 [[I_LS3]]
+; CHECK-NEXT: store i64 [[I_LS3]], ptr [[P_LS3]], align 4
+; CHECK-NEXT: [[I_NEXT_LS3]] = add i64 [[I_LS3]], 1
+; CHECK-NEXT: [[ITR_CHK7:%.*]] = icmp sle i64 [[I_NEXT_LS3]], [[TMP0]]
+; CHECK-NEXT: br i1 [[ITR_CHK7]], label %[[LOOP_LS3]], label %[[LS_EXIT3:.*]]
+; CHECK: [[LS_EXIT3]]:
+; CHECK-NEXT: br label %[[LS_FINAL_EXIT]]
+; CHECK: [[LS_FINAL_EXIT]]:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %loop
+
+loop:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %loop ]
+ %p = getelementptr i64, ptr %a, i64 %i
+ store i64 %i, ptr %p
+ %i.next = add i64 %i, 1
+ %c = icmp slt i64 %i.next, %n
+ br i1 %c, label %loop, label %exit
+
+exit:
+ ret void
+}
diff --git a/llvm/test/Transforms/LoopSplit/multiple-partitions.ll b/llvm/test/Transforms/LoopSplit/multiple-partitions.ll
new file mode 100644
index 0000000000000..74346767d7edd
--- /dev/null
+++ b/llvm/test/Transforms/LoopSplit/multiple-partitions.ll
@@ -0,0 +1,76 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt -passes='loop-simplify,lcssa,loop-split-test,verify' \
+; RUN: -loop-split-points=50,100 -S < %s | FileCheck %s
+
+; A loop split into three partitions: [0,49], [50,99], [100,n-1].
+
+define void @three_way(ptr %a, i64 %n) {
+; CHECK-LABEL: define void @three_way(
+; CHECK-SAME: ptr [[A:%.*]], i64 [[N:%.*]]) {
+; CHECK-NEXT: [[LS_GUARD0:.*:]]
+; CHECK-NEXT: [[SMAX:%.*]] = call i64 @llvm.smax.i64(i64 [[N]], i64 1)
+; CHECK-NEXT: [[TMP0:%.*]] = add nsw i64 [[SMAX]], -1
+; CHECK-NEXT: [[SMIN:%.*]] = call i64 @llvm.smin.i64(i64 [[TMP0]], i64 49)
+; CHECK-NEXT: [[SMIN1:%.*]] = call i64 @llvm.smin.i64(i64 [[TMP0]], i64 99)
+; CHECK-NEXT: [[ITR_CHK:%.*]] = icmp sle i64 0, [[SMIN]]
+; CHECK-NEXT: br i1 [[ITR_CHK]], label %[[ENTRY:.*]], label %[[LS_GUARD1:.*]]
+; CHECK: [[ENTRY]]:
+; CHECK-NEXT: br label %[[LOOP:.*]]
+; CHECK: [[LOOP]]:
+; CHECK-NEXT: [[I:%.*]] = phi i64 [ 0, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[LOOP]] ]
+; CHECK-NEXT: [[P:%.*]] = getelementptr i64, ptr [[A]], i64 [[I]]
+; CHECK-NEXT: store i64 [[I]], ptr [[P]], align 4
+; CHECK-NEXT: [[I_NEXT]] = add i64 [[I]], 1
+; CHECK-NEXT: [[ITR_CHK2:%.*]] = icmp sle i64 [[I_NEXT]], [[SMIN]]
+; CHECK-NEXT: br i1 [[ITR_CHK2]], label %[[LOOP]], label %[[EXIT:.*]]
+; CHECK: [[EXIT]]:
+; CHECK-NEXT: br label %[[LS_GUARD1]]
+; CHECK: [[LS_GUARD1]]:
+; CHECK-NEXT: [[ITR_CHK3:%.*]] = icmp sle i64 50, [[SMIN1]]
+; CHECK-NEXT: br i1 [[ITR_CHK3]], label %[[ENTRY_LS1:.*]], label %[[LS_GUARD2:.*]]
+; CHECK: [[ENTRY_LS1]]:
+; CHECK-NEXT: br label %[[LOOP_LS1:.*]]
+; CHECK: [[LOOP_LS1]]:
+; CHECK-NEXT: [[I_LS1:%.*]] = phi i64 [ 50, %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
+; CHECK-NEXT: [[P_LS1:%.*]] = getelementptr i64, ptr [[A]], i64 [[I_LS1]]
+; CHECK-NEXT: store i64 [[I_LS1]], ptr [[P_LS1]], align 4
+; CHECK-NEXT: [[I_NEXT_LS1]] = add i64 [[I_LS1]], 1
+; CHECK-NEXT: [[ITR_CHK4:%.*]] = icmp sle i64 [[I_NEXT_LS1]], [[SMIN1]]
+; CHECK-NEXT: br i1 [[ITR_CHK4]], label %[[LOOP_LS1]], label %[[LS_EXIT1:.*]]
+; CHECK: [[LS_EXIT1]]:
+; CHECK-NEXT: br label %[[LS_GUARD2]]
+; CHECK: [[LS_GUARD2]]:
+; CHECK-NEXT: [[ITR_CHK5:%.*]] = icmp sle i64 100, [[TMP0]]
+; CHECK-NEXT: br i1 [[ITR_CHK5]], label %[[ENTRY_LS2:.*]], label %[[LS_FINAL_EXIT:.*]]
+; CHECK: [[ENTRY_LS2]]:
+; CHECK-NEXT: br label %[[LOOP_LS2:.*]]
+; CHECK: [[LOOP_LS2]]:
+; CHECK-NEXT: [[I_LS2:%.*]] = phi i64 [ 100, %[[ENTRY_LS2]] ], [ [[I_NEXT_LS2:%.*]], %[[LOOP_LS2]] ]
+; CHECK-NEXT: [[P_LS2:%.*]] = getelementptr i64, ptr [[A]], i64 [[I_LS2]]
+; CHECK-NEXT: store i64 [[I_LS2]], ptr [[P_LS2]], align 4
+; CHECK-NEXT: [[I_NEXT_LS2]] = add i64 [[I_LS2]], 1
+; CHECK-NEXT: [[ITR_CHK6:%.*]] = icmp sle i64 [[I_NEXT_LS2]], [[TMP0]]
+; CHECK-NEXT: br i1 [[ITR_CHK6]], label %[[LOOP_LS2]], label %[[LS_EXIT2:.*]]
+; CHECK: [[LS_EXIT2]]:
+; CHECK-NEXT: br label %[[LS_FINAL_EXIT]]
+; CHECK: [[LS_FINAL_EXIT]]:
+; CHECK-NEXT: ret void
+;
+; Each partition is entered only if a runtime min/iteration check passes; the
+; min(partitionEnd, loopEnd) clamp is a signed-min intrinsic.
+; The original loop plus two clones cover [0,49], [50,99] and [100,n-1], and a
+; dedicated final-exit block joins all three partitions.
+entry:
+ br label %loop
+
+loop:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %loop ]
+ %p = getelementptr i64, ptr %a, i64 %i
+ store i64 %i, ptr %p
+ %i.next = add i64 %i, 1
+ %c = icmp slt i64 %i.next, %n
+ br i1 %c, label %loop, label %exit
+
+exit:
+ ret void
+}
diff --git a/llvm/test/Transforms/LoopSplit/optional-guard.ll b/llvm/test/Transforms/LoopSplit/optional-guard.ll
new file mode 100644
index 0000000000000..e3fbb39363710
--- /dev/null
+++ b/llvm/test/Transforms/LoopSplit/optional-guard.ll
@@ -0,0 +1,45 @@
+; RUN: opt -passes='loop-simplify,lcssa,loop-split-test,verify' \
+; RUN: -loop-split-points=4 -loop-split-unguarded=0 \
+; RUN: -verify-dom-info -verify-loop-info -S < %s | FileCheck %s
+
+; Per-partition entry guards are optional. The split loop is bottom-tested, so
+; by default every partition is guarded by a `Start <= End` check that skips a
+; zero-iteration partition. A caller that can prove a partition runs at least
+; once may omit its guard (here partition 0, via -loop-split-unguarded=0): that
+; guard then enters the sub-loop unconditionally and has no skip edge to the
+; next partition. Partition 1 keeps its guard. The dominator tree is maintained
+; incrementally for the missing skip edge, so -verify-dom-info must pass.
+
+define i64 @reduction(ptr %a, i64 %n) {
+; CHECK-LABEL: define i64 @reduction(
+; Partition 0's guard enters unconditionally - no entry icmp, no skip edge.
+; CHECK: ls.guard0:
+; CHECK: br label %[[ENTRY:.*]]
+; CHECK: [[ENTRY]]:
+; CHECK-NEXT: br label %[[LOOP:.*]]
+; The first sub-loop is the original, clamped to the partition-0 end.
+; CHECK: [[LOOP]]:
+; CHECK: br i1 {{.*}}, label %[[LOOP]], label %[[EXIT:.*]]
+; CHECK: [[EXIT]]:
+; CHECK-NEXT: br label %[[GUARD1:.*]]
+; Partition 1 keeps its conditional guard.
+; CHECK: [[GUARD1]]:
+; CHECK-NEXT: [[CHK1:%.*]] = icmp sle i64 4, {{.*}}
+; CHECK-NEXT: br i1 [[CHK1]], label %[[ENTRY1:.*]], label %[[FINAL:.*]]
+entry:
+ br label %loop
+
+loop:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %loop ]
+ %sum = phi i64 [ 0, %entry ], [ %sum.next, %loop ]
+ %p = getelementptr i64, ptr %a, i64 %i
+ %v = load i64, ptr %p
+ %sum.next = add i64 %sum, %v
+ %i.next = add i64 %i, 1
+ %c = icmp slt i64 %i.next, %n
+ br i1 %c, label %loop, label %exit
+
+exit:
+ %sum.lcssa = phi i64 [ %sum.next, %loop ]
+ ret i64 %sum.lcssa
+}
diff --git a/llvm/test/Transforms/LoopSplit/partition-value-map.ll b/llvm/test/Transforms/LoopSplit/partition-value-map.ll
new file mode 100644
index 0000000000000..75fe5fb236a81
--- /dev/null
+++ b/llvm/test/Transforms/LoopSplit/partition-value-map.ll
@@ -0,0 +1,44 @@
+; RUN: opt -passes='loop-simplify,lcssa,loop-split-test' \
+; RUN: -loop-split-points=4,8 -loop-split-print-partition-map \
+; RUN: -disable-output < %s 2>&1 | FileCheck %s
+
+; LoopSplitUtils preserves the original-to-clone value map for every partition
+; and exposes it through getPartitionValue(). Partition 0 reuses the original
+; loop (identity mapping); later partitions return the cloned counterpart, named
+; with a `.lsN` suffix. The latch compare (%c) is rewritten rather than cloned,
+; so it has no surviving counterpart and is omitted from every partition.
+
+define i32 @sum(ptr %a, i32 %n) {
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+ %acc = phi i32 [ 0, %entry ], [ %acc.next, %loop ]
+ %p = getelementptr inbounds i32, ptr %a, i32 %i
+ %v = load i32, ptr %p
+ %acc.next = add i32 %acc, %v
+ %i.next = add i32 %i, 1
+ %c = icmp slt i32 %i.next, %n
+ br i1 %c, label %loop, label %exit
+
+exit:
+ %lcssa = phi i32 [ %acc.next, %loop ]
+ ret i32 %lcssa
+}
+
+; CHECK: LS-MAP partition 0:
+; CHECK: i -> i
+; CHECK: acc -> acc
+; CHECK: acc.next -> acc.next
+; CHECK: i.next -> i.next
+; CHECK: LS-MAP partition 1:
+; CHECK: i -> i.ls1
+; CHECK: acc -> acc.ls1
+; CHECK: acc.next -> acc.next.ls1
+; CHECK: i.next -> i.next.ls1
+; CHECK: LS-MAP partition 2:
+; CHECK: i -> i.ls2
+; CHECK: acc -> acc.ls2
+; CHECK: acc.next -> acc.next.ls2
+; CHECK: i.next -> i.next.ls2
diff --git a/llvm/test/Transforms/LoopSplit/reduction.ll b/llvm/test/Transforms/LoopSplit/reduction.ll
new file mode 100644
index 0000000000000..5b7d2ac2eae52
--- /dev/null
+++ b/llvm/test/Transforms/LoopSplit/reduction.ll
@@ -0,0 +1,72 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt -passes='loop-simplify,lcssa,loop-split-test,verify' \
+; RUN: -loop-split-points=50 -S < %s | FileCheck %s
+
+; A reduction (live-out) is correctly threaded across the two partitions: the
+; second partition seeds its accumulator with the first partition's join value
+; and a dedicated final-exit block merges the per-partition results.
+
+define i64 @reduction(ptr %a, i64 %n) {
+; CHECK-LABEL: define i64 @reduction(
+; CHECK-SAME: ptr [[A:%.*]], i64 [[N:%.*]]) {
+; CHECK-NEXT: [[LS_GUARD0:.*]]:
+; CHECK-NEXT: [[SMAX:%.*]] = call i64 @llvm.smax.i64(i64 [[N]], i64 1)
+; CHECK-NEXT: [[TMP0:%.*]] = add nsw i64 [[SMAX]], -1
+; CHECK-NEXT: [[SMIN:%.*]] = call i64 @llvm.smin.i64(i64 [[TMP0]], i64 49)
+; CHECK-NEXT: [[ITR_CHK:%.*]] = icmp sle i64 0, [[SMIN]]
+; CHECK-NEXT: br i1 [[ITR_CHK]], label %[[ENTRY:.*]], label %[[LS_GUARD1:.*]]
+; CHECK: [[ENTRY]]:
+; CHECK-NEXT: br label %[[LOOP:.*]]
+; CHECK: [[LOOP]]:
+; CHECK-NEXT: [[I:%.*]] = phi i64 [ 0, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[LOOP]] ]
+; CHECK-NEXT: [[SUM:%.*]] = phi i64 [ 0, %[[ENTRY]] ], [ [[SUM_NEXT:%.*]], %[[LOOP]] ]
+; CHECK-NEXT: [[P:%.*]] = getelementptr i64, ptr [[A]], i64 [[I]]
+; CHECK-NEXT: [[V:%.*]] = load i64, ptr [[P]], align 4
+; CHECK-NEXT: [[SUM_NEXT]] = add i64 [[SUM]], [[V]]
+; CHECK-NEXT: [[I_NEXT]] = add i64 [[I]], 1
+; CHECK-NEXT: [[ITR_CHK1:%.*]] = icmp sle i64 [[I_NEXT]], [[SMIN]]
+; CHECK-NEXT: br i1 [[ITR_CHK1]], label %[[LOOP]], label %[[EXIT:.*]]
+; CHECK: [[EXIT]]:
+; CHECK-NEXT: br label %[[LS_GUARD1]]
+; CHECK: [[LS_GUARD1]]:
+; CHECK-NEXT: [[SUM_NEXT5:%.*]] = phi i64 [ [[SUM_NEXT]], %[[EXIT]] ], [ 0, %[[LS_GUARD0]] ]
+; CHECK-NEXT: [[ITR_CHK2:%.*]] = icmp sle i64 50, [[TMP0]]
+; CHECK-NEXT: br i1 [[ITR_CHK2]], label %[[ENTRY_LS1:.*]], label %[[LS_FINAL_EXIT:.*]]
+; CHECK: [[ENTRY_LS1]]:
+; CHECK-NEXT: br label %[[LOOP_LS1:.*]]
+; CHECK: [[LOOP_LS1]]:
+; CHECK-NEXT: [[I_LS1:%.*]] = phi i64 [ 50, %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
+; CHECK-NEXT: [[SUM_LS1:%.*]] = phi i64 [ [[SUM_NEXT5]], %[[ENTRY_LS1]] ], [ [[SUM_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
+; CHECK-NEXT: [[P_LS1:%.*]] = getelementptr i64, ptr [[A]], i64 [[I_LS1]]
+; CHECK-NEXT: [[V_LS1:%.*]] = load i64, ptr [[P_LS1]], align 4
+; CHECK-NEXT: [[SUM_NEXT_LS1]] = add i64 [[SUM_LS1]], [[V_LS1]]
+; CHECK-NEXT: [[I_NEXT_LS1]] = add i64 [[I_LS1]], 1
+; CHECK-NEXT: [[ITR_CHK3:%.*]] = icmp sle i64 [[I_NEXT_LS1]], [[TMP0]]
+; CHECK-NEXT: br i1 [[ITR_CHK3]], label %[[LOOP_LS1]], label %[[LS_EXIT1:.*]]
+; CHECK: [[LS_EXIT1]]:
+; CHECK-NEXT: br label %[[LS_FINAL_EXIT]]
+; CHECK: [[LS_FINAL_EXIT]]:
+; CHECK-NEXT: [[SUM_NEXT4:%.*]] = phi i64 [ [[SUM_NEXT_LS1]], %[[LS_EXIT1]] ], [ [[SUM_NEXT5]], %[[LS_GUARD1]] ]
+; CHECK-NEXT: ret i64 [[SUM_NEXT4]]
+;
+; The second partition's entry guard joins the first partition's result with the
+; chain-entry value (used when the first partition runs zero iterations).
+; The cloned partition seeds its accumulator from that join value.
+; A dedicated final-exit block merges the per-partition results.
+entry:
+ br label %loop
+
+loop:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %loop ]
+ %sum = phi i64 [ 0, %entry ], [ %sum.next, %loop ]
+ %p = getelementptr i64, ptr %a, i64 %i
+ %v = load i64, ptr %p
+ %sum.next = add i64 %sum, %v
+ %i.next = add i64 %i, 1
+ %c = icmp slt i64 %i.next, %n
+ br i1 %c, label %loop, label %exit
+
+exit:
+ %sum.lcssa = phi i64 [ %sum.next, %loop ]
+ ret i64 %sum.lcssa
+}
>From 92a3ccbd9fd061be79ed6a4a0625d8eabb5160eb Mon Sep 17 00:00:00 2001
From: Ashutosh Nema <ashu1212 at gmail.com>
Date: Mon, 13 Jul 2026 16:23:29 +0530
Subject: [PATCH 2/6] [Transforms][Utils] LoopSplitUtils: support constant-step
and top-tested loops
Generalize the induction analysis from a unit-step, bottom-tested latch compare
to any non-zero constant step in either direction, and to a counted exit
in the latch (bottom-tested) or header (top-tested). analyzeInduction() scans
the latch and header candidates, matches an InductionDescriptor PHI against the
exit compare, and records the signed step and exiting block. The induction
end comes from the utility's own counted end (with a one-step correction for
top-tested loops), the induction PHI is carried across partitions so a non-unit
stride tiles correctly, and the empty/clamp predicates follow the step and the
bottom/top-tested + PHI/step-compare combination. Adds the ExitingBlk plumbing
and getInductionEnd()/getPartitionLoop().
Refreshes the autogenerated CHECK lines of the existing LoopSplit tests to match
the new (behaviour-preserving, verifier-clean) output, and fixes descending-step
detection in the loop-split-test driver.
---
.../llvm/Transforms/Utils/LoopSplitUtils.h | 38 +++-
.../Transforms/Utils/LoopSplitTestPass.cpp | 6 +-
llvm/lib/Transforms/Utils/LoopSplitUtils.cpp | 178 ++++++++++++------
llvm/test/Transforms/LoopSplit/basic.ll | 5 +-
llvm/test/Transforms/LoopSplit/descending.ll | 5 +-
.../LoopSplit/empty-leading-partition.ll | 9 +-
.../Transforms/LoopSplit/four-partitions.ll | 11 +-
.../LoopSplit/multiple-partitions.ll | 8 +-
llvm/test/Transforms/LoopSplit/reduction.ll | 9 +-
9 files changed, 175 insertions(+), 94 deletions(-)
diff --git a/llvm/include/llvm/Transforms/Utils/LoopSplitUtils.h b/llvm/include/llvm/Transforms/Utils/LoopSplitUtils.h
index 1d68e8db5d774..e4878764b445f 100644
--- a/llvm/include/llvm/Transforms/Utils/LoopSplitUtils.h
+++ b/llvm/include/llvm/Transforms/Utils/LoopSplitUtils.h
@@ -14,6 +14,7 @@
#ifndef LLVM_TRANSFORMS_UTILS_LOOPSPLITUTILS_H
#define LLVM_TRANSFORMS_UTILS_LOOPSPLITUTILS_H
+#include "llvm/ADT/APInt.h"
#include "llvm/ADT/SmallVector.h"
#include "llvm/Support/Compiler.h"
#include "llvm/Transforms/Utils/ValueMapper.h"
@@ -49,14 +50,18 @@ class LoopSplitUtils {
LoopSplitUtils(Loop *L, LoopInfo *LI, ScalarEvolution *SE, DominatorTree *DT)
: L(L), LI(LI), SE(SE), DT(DT) {}
- /// Analyze \p L and return true if it is a counted loop this utility can split:
- /// a bottom-tested single-exit loop in LCSSA form with a unique unit-step
- /// integer induction and a computable trip count. Must succeed before split().
+ /// Analyze \p L; true if it is a counted loop we can split: LCSSA, a counted
+ /// exit in the latch or header, a constant-step integer induction, and an
+ /// exact count. Extra side exits are allowed. Must succeed before split().
LLVM_ABI bool isLegal();
/// Return the loop's induction variable. Valid only after isLegal() succeeds.
PHINode *getInductionVariable() const { return Induction; }
+ /// Induction value on the final counted iteration (inclusive) -- the end of
+ /// the space the partitions tile. Valid only after isLegal() succeeds.
+ const SCEV *getInductionEnd() const { return InductionEnd; }
+
/// Append an inclusive partition range [Start, End] in iteration order.
/// Partitions must tile the whole space: first Start = induction start, each
/// later Start = previous End +/- step, last End = induction end (desc: S >= E).
@@ -75,6 +80,10 @@ class LoopSplitUtils {
unsigned getNumPartitions() const { return Partitions.size(); }
+ /// The loop driving partition \p PartitionIndex: the original loop for 0, the
+ /// clone for a later partition, or null if not cloned. Valid after split().
+ LLVM_ABI Loop *getPartitionLoop(unsigned PartitionIndex) const;
+
/// Perform the split. Requires a successful isLegal() and at least two
/// partitions. Returns true if the loop was rewritten.
LLVM_ABI bool split();
@@ -108,8 +117,10 @@ class LoopSplitUtils {
BasicBlock *GuardBlock = nullptr;
BasicBlock *Preheader = nullptr;
BasicBlock *Exit = nullptr;
+ BasicBlock *ExitingBlk =
+ nullptr; // block holding this partition's exit test.
Loop *SubLoop = nullptr;
- Value *LatchIndOp = nullptr; // induction operand of the latch compare.
+ Value *LatchIndOp = nullptr; // induction operand of the exit-test compare.
};
/// Per-split() scratch threaded through the phase helpers (the escaping
@@ -124,11 +135,15 @@ class LoopSplitUtils {
// Induction analysis, populated by isLegal().
PHINode *Induction = nullptr;
- ICmpInst *LatchCmp = nullptr; // the loop's latch exit compare.
- Value *LatchIndOperand = nullptr; // induction operand of the latch compare.
- bool LatchUsesInductionPHI = false; // latch compares the PHI, not the step.
+ BasicBlock *ExitingBlock = nullptr; // the loop's single exiting block.
+ bool IsTopTested = false; // exit test precedes the body (header).
+ ICmpInst *LatchCmp = nullptr; // the exiting block's exit compare.
+ Value *LatchIndOperand = nullptr; // induction operand of the exit compare.
+ bool LatchUsesInductionPHI =
+ false; // exit test compares the PHI, not the step.
bool InductionIsSigned = false; // iteration ordering signedness.
- bool InductionIsDescending = false; // step is -1 (loop counts down).
+ bool InductionIsDescending = false; // step is negative (loop counts down).
+ APInt InductionStep; // signed constant step, induction width.
const SCEV *InductionEnd = nullptr;
/// One record per partition, in add order.
@@ -154,8 +169,11 @@ class LoopSplitUtils {
void chainPartitions(SplitState &S);
/// Rebuild SSA for every escaping value with a per-value SSAUpdater.
void reconstructSSA(SplitState &S);
- /// Clamp \p PL's latch so it iterates only within [start, \p SelEnd].
- void rewriteLatch(Loop *PL, Value *IndOp, Value *SelEnd, BasicBlock *Exit);
+ /// Clamp \p PL's exit test (in \p ExitingBlk) so it iterates only within
+ /// [start, \p SelEnd], sending the out-of-range edge to \p Exit.
+ void rewriteLatch(Loop *PL, BasicBlock *ExitingBlk, Value *IndOp,
+ Value *SelEnd, BasicBlock *Exit,
+ bool IsLastPartition = false);
};
} // namespace llvm
diff --git a/llvm/lib/Transforms/Utils/LoopSplitTestPass.cpp b/llvm/lib/Transforms/Utils/LoopSplitTestPass.cpp
index b94ab1ee82496..f650cac65192b 100644
--- a/llvm/lib/Transforms/Utils/LoopSplitTestPass.cpp
+++ b/llvm/lib/Transforms/Utils/LoopSplitTestPass.cpp
@@ -63,8 +63,8 @@ static bool splitLoop(Loop *L, ScalarEvolution &SE, DominatorTree &DT,
return false;
const SCEV *Start = IndAR->getStart();
- const SCEV *BTC = SE.getBackedgeTakenCount(L);
- const SCEV *End = IndAR->evaluateAtIteration(BTC, SE);
+ // Use the utility's counted induction end (works for multi-exit loops too).
+ const SCEV *End = LSU.getInductionEnd();
Type *Ty = Start->getType();
if (End->getType() != Ty)
End = SE.getTruncateExpr(End, Ty);
@@ -74,7 +74,7 @@ static bool splitLoop(Loop *L, ScalarEvolution &SE, DominatorTree &DT,
// iteration `Start +/- offset`; the previous partition ends one step before.
bool Descending = false;
if (const auto *StepC = dyn_cast<SCEVConstant>(IndAR->getStepRecurrence(SE)))
- Descending = StepC->getValue()->isMinusOne();
+ Descending = StepC->getAPInt().isNegative();
const SCEV *PrevStart = Start;
const SCEV *One = SE.getOne(Ty);
diff --git a/llvm/lib/Transforms/Utils/LoopSplitUtils.cpp b/llvm/lib/Transforms/Utils/LoopSplitUtils.cpp
index 54a3d616f902f..fb4b4b8858d55 100644
--- a/llvm/lib/Transforms/Utils/LoopSplitUtils.cpp
+++ b/llvm/lib/Transforms/Utils/LoopSplitUtils.cpp
@@ -48,6 +48,7 @@
#include "llvm/Transforms/Utils/LoopSplitUtils.h"
#include "llvm/ADT/DenseMap.h"
+#include "llvm/Analysis/IVDescriptors.h"
#include "llvm/Analysis/LoopInfo.h"
#include "llvm/Analysis/ScalarEvolution.h"
#include "llvm/Analysis/ScalarEvolutionExpressions.h"
@@ -124,6 +125,12 @@ void LoopSplitUtils::avoidPartitionGuard(unsigned PartitionIndex) {
Partitions[PartitionIndex].Guarded = false;
}
+// Return the loop driving a partition (original for 0, clone otherwise).
+Loop *LoopSplitUtils::getPartitionLoop(unsigned PartitionIndex) const {
+ assert(PartitionIndex < getNumPartitions() && "partition index out of range");
+ return Partitions[PartitionIndex].SubLoop;
+}
+
// Return a partition's original-to-clone map, or null if it has none.
const ValueToValueMapTy *
LoopSplitUtils::getPartitionValueMap(unsigned PartitionIndex) const {
@@ -149,41 +156,64 @@ Value *LoopSplitUtils::getPartitionValue(const Value *V,
// Find the induction variable and the latch operand it is compared against;
// returns the induction's add-recurrence, or null if the loop is unsuitable.
const SCEVAddRecExpr *LoopSplitUtils::analyzeInduction() {
- // The loop must exit on an integer compare living in the latch.
- LatchCmp = L->getLatchCmpInst();
- if (!LatchCmp || LatchCmp->getParent() != L->getLoopLatch())
- return nullptr;
+ BasicBlock *Header = L->getHeader();
+ BasicBlock *Latch = L->getLoopLatch();
- // SCEV's induction variable, restricted to a unit-step affine recurrence.
- Induction = L->getInductionVariable(*SE);
- if (!Induction)
- return nullptr;
- const auto *AR = dyn_cast<SCEVAddRecExpr>(SE->getSCEV(Induction));
- if (!AR || !AR->isAffine())
- return nullptr;
- // Accept a unit step in either direction: +1 (ascending) or -1 (descending).
- const auto *Step = dyn_cast<SCEVConstant>(AR->getStepRecurrence(*SE));
- if (!Step || !(Step->getValue()->isOne() || Step->getValue()->isMinusOne()))
- return nullptr;
- InductionIsDescending = Step->getValue()->isMinusOne();
+ // The counted exit lives in the latch (bottom-tested) or header (top-tested).
+ // Try the latch first so a single-exit loop keeps its previous classification.
+ SmallVector<BasicBlock *, 2> Candidates;
+ Candidates.push_back(Latch);
+ if (Header != Latch)
+ Candidates.push_back(Header);
- // The induction's "next" value (i + 1), produced in the latch.
- auto *StepInst = dyn_cast<Instruction>(
- Induction->getIncomingValueForBlock(L->getLoopLatch()));
- if (!StepInst)
- return nullptr;
+ for (BasicBlock *Cand : Candidates) {
+ if (!L->isLoopExiting(Cand))
+ continue;
+ auto *ExitBr = dyn_cast<BranchInst>(Cand->getTerminator());
+ if (!ExitBr || !ExitBr->isConditional())
+ continue;
+ auto *Cmp = dyn_cast<ICmpInst>(ExitBr->getCondition());
+ if (!Cmp)
+ continue;
- // Select the compare operand that is the induction (PHI or its step).
- if (LatchCmp->getOperand(0) == Induction ||
- LatchCmp->getOperand(0) == StepInst)
- LatchIndOperand = LatchCmp->getOperand(0);
- else if (LatchCmp->getOperand(1) == Induction ||
- LatchCmp->getOperand(1) == StepInst)
- LatchIndOperand = LatchCmp->getOperand(1);
- else
- return nullptr;
- LatchUsesInductionPHI = (LatchIndOperand == Induction);
- return AR;
+ // The exit compare identifies the induction: one operand is the header PHI
+ // (pre-increment) or its PHI+step value (post-increment), the other the
+ // invariant bound. Accept any non-zero constant step, either direction.
+ Value *CmpOp0 = Cmp->getOperand(0), *CmpOp1 = Cmp->getOperand(1);
+ for (PHINode &PN : Header->phis()) {
+ if (!PN.getType()->isIntegerTy() || !SE->isSCEVable(PN.getType()))
+ continue;
+ InductionDescriptor ID;
+ if (!InductionDescriptor::isInductionPHI(&PN, L, SE, ID))
+ continue;
+ const auto *Step = dyn_cast<SCEVConstant>(ID.getStep());
+ if (!Step || Step->getValue()->isZero())
+ continue;
+
+ // Match a compare operand to this IV: the PHI is the pre-increment value,
+ // its latch-incoming value is the post-increment (PHI+step) value.
+ Value *StepVal = PN.getIncomingValueForBlock(Latch);
+ bool UsesPHI;
+ if (CmpOp0 == &PN || CmpOp1 == &PN)
+ UsesPHI = true;
+ else if (CmpOp0 == StepVal || CmpOp1 == StepVal)
+ UsesPHI = false;
+ else
+ continue;
+
+ // Commit the counted-exit analysis for this candidate.
+ ExitingBlock = Cand;
+ IsTopTested = Cand != Latch;
+ LatchCmp = Cmp;
+ LatchIndOperand = UsesPHI ? static_cast<Value *>(&PN) : StepVal;
+ LatchUsesInductionPHI = UsesPHI;
+ Induction = &PN;
+ InductionStep = Step->getAPInt();
+ InductionIsDescending = InductionStep.isNegative();
+ return cast<SCEVAddRecExpr>(SE->getSCEV(&PN));
+ }
+ }
+ return nullptr;
}
// Decide whether the iteration ordering is signed or unsigned.
@@ -211,9 +241,8 @@ bool LoopSplitUtils::isLegal() {
LLVM_DEBUG(dbgs() << "LS: missing preheader/latch\n");
return false;
}
- if (!L->getExitingBlock() || !L->getExitBlock() ||
- L->getExitingBlock() != L->getLoopLatch()) {
- LLVM_DEBUG(dbgs() << "LS: not a bottom-tested single-exit loop\n");
+ if (!L->getExitingBlock() || !L->getExitBlock()) {
+ LLVM_DEBUG(dbgs() << "LS: not a single-exit loop\n");
return false;
}
if (!L->isLCSSAForm(*DT)) {
@@ -221,23 +250,30 @@ bool LoopSplitUtils::isLegal() {
return false;
}
- // A computable backedge-taken count fixes the iteration space we rebuild.
- const SCEV *BTC = SE->getBackedgeTakenCount(L);
- if (isa<SCEVCouldNotCompute>(BTC)) {
- LLVM_DEBUG(dbgs() << "LS: no computable trip count\n");
- return false;
- }
-
+ // Identify the counted (induction) exit.
const SCEVAddRecExpr *IndAR = analyzeInduction();
if (!IndAR) {
- LLVM_DEBUG(dbgs() << "LS: no unique unit-step integer induction\n");
+ LLVM_DEBUG(dbgs() << "LS: no counted induction exit\n");
return false;
}
if (!computeSignedness(IndAR))
return false;
+ // A computable backedge-taken count fixes the iteration space we rebuild.
+ const SCEV *BTC = SE->getBackedgeTakenCount(L);
+ if (isa<SCEVCouldNotCompute>(BTC)) {
+ LLVM_DEBUG(dbgs() << "LS: no computable trip count\n");
+ return false;
+ }
+
+ // evaluateAtIteration(BTC) is the induction value at the final exit test: the
+ // last value the body ran with when bottom-tested; for top-tested the body
+ // ran a step earlier, so step back to the last executed value.
InductionEnd = IndAR->evaluateAtIteration(BTC, *SE);
+ if (IsTopTested)
+ InductionEnd =
+ SE->getMinusSCEV(InductionEnd, IndAR->getStepRecurrence(*SE));
// Start and end must share the induction type; reject any width mismatch.
if (InductionEnd->getType() != IndAR->getStart()->getType()) {
LLVM_DEBUG(dbgs() << "LS: induction end/start type mismatch\n");
@@ -317,16 +353,19 @@ void LoopSplitUtils::collectEscapingValues(SplitState &S) {
// predecessor is the original exit block.
DT->addNewBlock(S.FinalExit, S.ExitBlock);
- // (1) Carried values: each non-induction header PHI whose backedge value
- // differs from its initial value must resume in later partitions.
+ // (1) Carried values: each header PHI whose backedge value differs from its
+ // initial value must resume in later partitions. The induction PHI is now
+ // included -- carrying its runtime value lets a non-unit stride tile correctly.
DenseMap<Value *, unsigned> CarriedDefToEscapingIdx;
for (PHINode &HeaderPHI : L->getHeader()->phis()) {
- if (&HeaderPHI == Induction)
- continue;
- Value *CarriedValue = HeaderPHI.getIncomingValueForBlock(Latch);
+ Value *BackedgeValue = HeaderPHI.getIncomingValueForBlock(Latch);
Value *InitialValue = HeaderPHI.getIncomingValueForBlock(S.OrigPreheader);
- if (CarriedValue == InitialValue)
+ if (BackedgeValue == InitialValue)
continue; // invariant and equal to the initial value: nothing to carry.
+ // Value live at a partition's exit, seeding the next. Bottom-tested exits
+ // from the latch (backedge value live); top-tested exits from the header
+ // before the latch, so the header PHI holds it (and dominates the exit).
+ Value *CarriedValue = IsTopTested ? &HeaderPHI : BackedgeValue;
auto &EV = S.addEscaping(CarriedValue);
EV.CarriedHeaderPHI = &HeaderPHI;
// Track in-loop carried defs so a matching live-out in (2) merges onto
@@ -389,12 +428,12 @@ void LoopSplitUtils::expandPartitionBounds(SplitState &S) {
for (unsigned I = 0; I < N; ++I) {
PartitionInfo &P = Partitions[I];
- // Provably empty when Start overshoots End by exactly one step. Compile-time
- // only: a runtime overshoot wraps at the type extreme and would falsely enter.
+ // Provably empty when Start overshoots End by one step (Start - End == step).
+ // Compile-time only: a runtime overshoot wraps and would falsely enter.
const SCEV *PartWidth = SE->getMinusSCEV(P.StartExpr, P.EndExpr);
if (auto *PartWidthConst = dyn_cast<SCEVConstant>(PartWidth)) {
const APInt &W = PartWidthConst->getAPInt();
- P.Empty = InductionIsDescending ? W.isAllOnes() : W.isOne();
+ P.Empty = W == InductionStep;
}
P.StartVal = Expander.expandCodeFor(P.StartExpr, IndTy, EntryGuardTerm);
@@ -424,6 +463,7 @@ void LoopSplitUtils::clonePartitions(SplitState &S) {
P0.GuardBlock = S.EntryGuard;
P0.Preheader = S.OrigPreheader;
P0.Exit = S.ExitBlock;
+ P0.ExitingBlk = ExitingBlock;
P0.SubLoop = L;
P0.LatchIndOp = LatchIndOperand;
@@ -458,6 +498,7 @@ void LoopSplitUtils::clonePartitions(SplitState &S) {
P.GuardBlock = Guardi;
P.Preheader = PHi;
P.Exit = Exiti;
+ P.ExitingBlk = cast<BasicBlock>(remapValue(VMap, ExitingBlock));
P.SubLoop = PL;
P.LatchIndOp = remapValue(VMap, LatchIndOperand);
@@ -469,21 +510,31 @@ void LoopSplitUtils::clonePartitions(SplitState &S) {
}
}
-// Replace a partition's latch test so it iterates only within [start, SelEnd].
-void LoopSplitUtils::rewriteLatch(Loop *PL, Value *IndOp, Value *SelEnd,
- BasicBlock *Exit) {
- auto *Term = cast<CondBrInst>(PL->getLoopLatch()->getTerminator());
+// Replace a partition's exit test so it iterates only within [start, SelEnd].
+void LoopSplitUtils::rewriteLatch(Loop *PL, BasicBlock *ExitingBlk,
+ Value *IndOp, Value *SelEnd, BasicBlock *Exit,
+ bool IsLastPartition) {
+ auto *Term = cast<BranchInst>(ExitingBlk->getTerminator());
auto *Cmp = cast<ICmpInst>(Term->getCondition());
+ // Preserve the original in-loop successor (the header for a bottom-tested
+ // loop); only the out-of-loop edge is redirected to this partition's exit.
+ BasicBlock *ContinueTarget = PL->contains(Term->getSuccessor(0))
+ ? Term->getSuccessor(0)
+ : Term->getSuccessor(1);
IRBuilder<> B(Cmp);
+
Value *Bound = SelEnd;
if (Bound->getType() != IndOp->getType())
Bound = B.CreateIntCast(Bound, IndOp->getType(), InductionIsSigned);
- // Strict when the PHI itself is compared, inclusive when the step value is.
- ICmpInst::Predicate Pred = continuePredicate(
- InductionIsSigned, InductionIsDescending, /*Inclusive=*/!LatchUsesInductionPHI);
+ // The clamp keeps iterations with induction <= SelEnd. The continue test is
+ // inclusive iff (compares PHI) == (test precedes body) -- so bottom-tested+PHI
+ // and top-tested+step are strict, the other two inclusive.
+ bool Inclusive = LatchUsesInductionPHI == IsTopTested;
+ ICmpInst::Predicate Pred =
+ continuePredicate(InductionIsSigned, InductionIsDescending, Inclusive);
Value *NewCmp = B.CreateICmp(Pred, IndOp, Bound, "itr.chk");
B.SetInsertPoint(Term);
- B.CreateCondBr(NewCmp, PL->getHeader(), Exit);
+ B.CreateCondBr(NewCmp, ContinueTarget, Exit);
Term->eraseFromParent();
if (Cmp->use_empty())
Cmp->eraseFromParent();
@@ -532,7 +583,8 @@ void LoopSplitUtils::chainPartitions(SplitState &S) {
}
GuardTerm->eraseFromParent();
- rewriteLatch(P.SubLoop, P.LatchIndOp, P.SelEnd, P.Exit);
+ rewriteLatch(P.SubLoop, P.ExitingBlk, P.LatchIndOp, P.SelEnd, P.Exit,
+ /*IsLastPartition=*/I + 1 == N);
P.Exit->getTerminator()->setSuccessor(0, MergeAfter);
}
@@ -547,7 +599,9 @@ void LoopSplitUtils::chainPartitions(SplitState &S) {
PartitionInfo &Cur = Partitions[I];
DT->addNewBlock(Cur.GuardBlock, MergeTargetIDom(Prev));
DT->changeImmediateDominator(Cur.Preheader, Cur.GuardBlock);
- DT->addNewBlock(Cur.Exit, Cur.SubLoop->getLoopLatch());
+ // The partition's exit is reached only from its exiting block, so that block
+ // is its immediate dominator.
+ DT->addNewBlock(Cur.Exit, Cur.ExitingBlk);
}
// The final exit is the last partition's merge target.
DT->changeImmediateDominator(S.FinalExit, MergeTargetIDom(Partitions.back()));
diff --git a/llvm/test/Transforms/LoopSplit/basic.ll b/llvm/test/Transforms/LoopSplit/basic.ll
index 48e836bdb7cb0..3f354e17d929d 100644
--- a/llvm/test/Transforms/LoopSplit/basic.ll
+++ b/llvm/test/Transforms/LoopSplit/basic.ll
@@ -7,7 +7,7 @@
define void @basic(ptr %a, i64 %n) {
; CHECK-LABEL: define void @basic(
; CHECK-SAME: ptr [[A:%.*]], i64 [[N:%.*]]) {
-; CHECK-NEXT: [[LS_GUARD0:.*:]]
+; CHECK-NEXT: [[LS_GUARD0:.*]]:
; CHECK-NEXT: [[SMAX:%.*]] = call i64 @llvm.smax.i64(i64 [[N]], i64 1)
; CHECK-NEXT: [[TMP0:%.*]] = add nsw i64 [[SMAX]], -1
; CHECK-NEXT: [[SMIN:%.*]] = call i64 @llvm.smin.i64(i64 [[TMP0]], i64 49)
@@ -25,12 +25,13 @@ define void @basic(ptr %a, i64 %n) {
; CHECK: [[EXIT]]:
; CHECK-NEXT: br label %[[LS_GUARD1]]
; CHECK: [[LS_GUARD1]]:
+; CHECK-NEXT: [[I_NEXT4:%.*]] = phi i64 [ [[I_NEXT]], %[[EXIT]] ], [ 0, %[[LS_GUARD0]] ]
; CHECK-NEXT: [[ITR_CHK2:%.*]] = icmp sle i64 50, [[TMP0]]
; CHECK-NEXT: br i1 [[ITR_CHK2]], label %[[ENTRY_LS1:.*]], label %[[LS_FINAL_EXIT:.*]]
; CHECK: [[ENTRY_LS1]]:
; CHECK-NEXT: br label %[[LOOP_LS1:.*]]
; CHECK: [[LOOP_LS1]]:
-; CHECK-NEXT: [[I_LS1:%.*]] = phi i64 [ 50, %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
+; CHECK-NEXT: [[I_LS1:%.*]] = phi i64 [ [[I_NEXT4]], %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
; CHECK-NEXT: [[P_LS1:%.*]] = getelementptr i64, ptr [[A]], i64 [[I_LS1]]
; CHECK-NEXT: store i64 [[I_LS1]], ptr [[P_LS1]], align 4
; CHECK-NEXT: [[I_NEXT_LS1]] = add i64 [[I_LS1]], 1
diff --git a/llvm/test/Transforms/LoopSplit/descending.ll b/llvm/test/Transforms/LoopSplit/descending.ll
index e6e1b56acd5f5..9c0612315d6c5 100644
--- a/llvm/test/Transforms/LoopSplit/descending.ll
+++ b/llvm/test/Transforms/LoopSplit/descending.ll
@@ -8,7 +8,7 @@
define void @descending(ptr %a, i64 %n) {
; CHECK-LABEL: define void @descending(
; CHECK-SAME: ptr [[A:%.*]], i64 [[N:%.*]]) {
-; CHECK-NEXT: [[LS_GUARD0:.*:]]
+; CHECK-NEXT: [[LS_GUARD0:.*]]:
; CHECK-NEXT: [[TMP0:%.*]] = add i64 [[N]], -49
; CHECK-NEXT: [[TMP1:%.*]] = add i64 [[N]], -1
; CHECK-NEXT: [[SMIN:%.*]] = call i64 @llvm.smin.i64(i64 [[TMP1]], i64 -1)
@@ -29,12 +29,13 @@ define void @descending(ptr %a, i64 %n) {
; CHECK: [[EXIT]]:
; CHECK-NEXT: br label %[[LS_GUARD1]]
; CHECK: [[LS_GUARD1]]:
+; CHECK-NEXT: [[I_NEXT4:%.*]] = phi i64 [ [[I_NEXT]], %[[EXIT]] ], [ [[N]], %[[LS_GUARD0]] ]
; CHECK-NEXT: [[ITR_CHK2:%.*]] = icmp sge i64 [[TMP3]], [[TMP2]]
; CHECK-NEXT: br i1 [[ITR_CHK2]], label %[[ENTRY_LS1:.*]], label %[[LS_FINAL_EXIT:.*]]
; CHECK: [[ENTRY_LS1]]:
; CHECK-NEXT: br label %[[LOOP_LS1:.*]]
; CHECK: [[LOOP_LS1]]:
-; CHECK-NEXT: [[I_LS1:%.*]] = phi i64 [ [[TMP3]], %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
+; CHECK-NEXT: [[I_LS1:%.*]] = phi i64 [ [[I_NEXT4]], %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
; CHECK-NEXT: [[P_LS1:%.*]] = getelementptr i64, ptr [[A]], i64 [[I_LS1]]
; CHECK-NEXT: store i64 [[I_LS1]], ptr [[P_LS1]], align 4
; CHECK-NEXT: [[I_NEXT_LS1]] = add i64 [[I_LS1]], -1
diff --git a/llvm/test/Transforms/LoopSplit/empty-leading-partition.ll b/llvm/test/Transforms/LoopSplit/empty-leading-partition.ll
index dd0660a43c83d..ad9c5bfb2586c 100644
--- a/llvm/test/Transforms/LoopSplit/empty-leading-partition.ll
+++ b/llvm/test/Transforms/LoopSplit/empty-leading-partition.ll
@@ -30,14 +30,15 @@ define i64 @reduction(ptr %a, i64 %n) {
; CHECK: [[EXIT]]:
; CHECK-NEXT: br label %[[LS_GUARD1]]
; CHECK: [[LS_GUARD1]]:
-; CHECK-NEXT: [[SUM_NEXT4:%.*]] = phi i64 [ [[SUM_NEXT]], %[[EXIT]] ], [ 0, %[[LS_GUARD0]] ]
+; CHECK-NEXT: [[SUM_NEXT5:%.*]] = phi i64 [ [[SUM_NEXT]], %[[EXIT]] ], [ 0, %[[LS_GUARD0]] ]
+; CHECK-NEXT: [[I_NEXT3:%.*]] = phi i64 [ [[I_NEXT]], %[[EXIT]] ], [ 0, %[[LS_GUARD0]] ]
; CHECK-NEXT: [[ITR_CHK1:%.*]] = icmp sle i64 0, [[TMP0]]
; CHECK-NEXT: br i1 [[ITR_CHK1]], label %[[ENTRY_LS1:.*]], label %[[LS_FINAL_EXIT:.*]]
; CHECK: [[ENTRY_LS1]]:
; CHECK-NEXT: br label %[[LOOP_LS1:.*]]
; CHECK: [[LOOP_LS1]]:
-; CHECK-NEXT: [[I_LS1:%.*]] = phi i64 [ 0, %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
-; CHECK-NEXT: [[SUM_LS1:%.*]] = phi i64 [ [[SUM_NEXT4]], %[[ENTRY_LS1]] ], [ [[SUM_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
+; CHECK-NEXT: [[I_LS1:%.*]] = phi i64 [ [[I_NEXT3]], %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
+; CHECK-NEXT: [[SUM_LS1:%.*]] = phi i64 [ [[SUM_NEXT5]], %[[ENTRY_LS1]] ], [ [[SUM_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
; CHECK-NEXT: [[P_LS1:%.*]] = getelementptr i64, ptr [[A]], i64 [[I_LS1]]
; CHECK-NEXT: [[V_LS1:%.*]] = load i64, ptr [[P_LS1]], align 4
; CHECK-NEXT: [[SUM_NEXT_LS1]] = add i64 [[SUM_LS1]], [[V_LS1]]
@@ -47,7 +48,7 @@ define i64 @reduction(ptr %a, i64 %n) {
; CHECK: [[LS_EXIT1]]:
; CHECK-NEXT: br label %[[LS_FINAL_EXIT]]
; CHECK: [[LS_FINAL_EXIT]]:
-; CHECK-NEXT: [[SUM_NEXT3:%.*]] = phi i64 [ [[SUM_NEXT_LS1]], %[[LS_EXIT1]] ], [ [[SUM_NEXT4]], %[[LS_GUARD1]] ]
+; CHECK-NEXT: [[SUM_NEXT3:%.*]] = phi i64 [ [[SUM_NEXT_LS1]], %[[LS_EXIT1]] ], [ [[SUM_NEXT5]], %[[LS_GUARD1]] ]
; CHECK-NEXT: ret i64 [[SUM_NEXT3]]
;
; The empty leading partition is never entered: an unconditional fall-through.
diff --git a/llvm/test/Transforms/LoopSplit/four-partitions.ll b/llvm/test/Transforms/LoopSplit/four-partitions.ll
index 4ebd030691f2d..1e49370d9d93b 100644
--- a/llvm/test/Transforms/LoopSplit/four-partitions.ll
+++ b/llvm/test/Transforms/LoopSplit/four-partitions.ll
@@ -9,7 +9,7 @@
define void @four(ptr %a, i64 %n) {
; CHECK-LABEL: define void @four(
; CHECK-SAME: ptr [[A:%.*]], i64 [[N:%.*]]) {
-; CHECK-NEXT: [[LS_GUARD0:.*:]]
+; CHECK-NEXT: [[LS_GUARD0:.*]]:
; CHECK-NEXT: [[SMAX:%.*]] = call i64 @llvm.smax.i64(i64 [[N]], i64 1)
; CHECK-NEXT: [[TMP0:%.*]] = add nsw i64 [[SMAX]], -1
; CHECK-NEXT: [[SMIN2:%.*]] = call i64 @llvm.smin.i64(i64 [[TMP0]], i64 99)
@@ -29,12 +29,13 @@ define void @four(ptr %a, i64 %n) {
; CHECK: [[EXIT]]:
; CHECK-NEXT: br label %[[LS_GUARD1]]
; CHECK: [[LS_GUARD1]]:
+; CHECK-NEXT: [[I_NEXT10:%.*]] = phi i64 [ [[I_NEXT]], %[[EXIT]] ], [ 0, %[[LS_GUARD0]] ]
; CHECK-NEXT: [[ITR_CHK2:%.*]] = icmp sle i64 100, [[SMIN]]
; CHECK-NEXT: br i1 [[ITR_CHK2]], label %[[ENTRY_LS1:.*]], label %[[LS_GUARD2:.*]]
; CHECK: [[ENTRY_LS1]]:
; CHECK-NEXT: br label %[[LOOP_LS1:.*]]
; CHECK: [[LOOP_LS1]]:
-; CHECK-NEXT: [[I_LS1:%.*]] = phi i64 [ 100, %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
+; CHECK-NEXT: [[I_LS1:%.*]] = phi i64 [ [[I_NEXT10]], %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
; CHECK-NEXT: [[P_LS1:%.*]] = getelementptr i64, ptr [[A]], i64 [[I_LS1]]
; CHECK-NEXT: store i64 [[I_LS1]], ptr [[P_LS1]], align 4
; CHECK-NEXT: [[I_NEXT_LS1]] = add i64 [[I_LS1]], 1
@@ -43,12 +44,13 @@ define void @four(ptr %a, i64 %n) {
; CHECK: [[LS_EXIT1]]:
; CHECK-NEXT: br label %[[LS_GUARD2]]
; CHECK: [[LS_GUARD2]]:
+; CHECK-NEXT: [[I_NEXT11:%.*]] = phi i64 [ [[I_NEXT_LS1]], %[[LS_EXIT1]] ], [ [[I_NEXT10]], %[[LS_GUARD1]] ]
; CHECK-NEXT: [[ITR_CHK4:%.*]] = icmp sle i64 200, [[SMIN1]]
; CHECK-NEXT: br i1 [[ITR_CHK4]], label %[[ENTRY_LS2:.*]], label %[[LS_GUARD3:.*]]
; CHECK: [[ENTRY_LS2]]:
; CHECK-NEXT: br label %[[LOOP_LS2:.*]]
; CHECK: [[LOOP_LS2]]:
-; CHECK-NEXT: [[I_LS2:%.*]] = phi i64 [ 200, %[[ENTRY_LS2]] ], [ [[I_NEXT_LS2:%.*]], %[[LOOP_LS2]] ]
+; CHECK-NEXT: [[I_LS2:%.*]] = phi i64 [ [[I_NEXT11]], %[[ENTRY_LS2]] ], [ [[I_NEXT_LS2:%.*]], %[[LOOP_LS2]] ]
; CHECK-NEXT: [[P_LS2:%.*]] = getelementptr i64, ptr [[A]], i64 [[I_LS2]]
; CHECK-NEXT: store i64 [[I_LS2]], ptr [[P_LS2]], align 4
; CHECK-NEXT: [[I_NEXT_LS2]] = add i64 [[I_LS2]], 1
@@ -57,12 +59,13 @@ define void @four(ptr %a, i64 %n) {
; CHECK: [[LS_EXIT2]]:
; CHECK-NEXT: br label %[[LS_GUARD3]]
; CHECK: [[LS_GUARD3]]:
+; CHECK-NEXT: [[I_NEXT12:%.*]] = phi i64 [ [[I_NEXT_LS2]], %[[LS_EXIT2]] ], [ [[I_NEXT11]], %[[LS_GUARD2]] ]
; CHECK-NEXT: [[ITR_CHK6:%.*]] = icmp sle i64 300, [[TMP0]]
; CHECK-NEXT: br i1 [[ITR_CHK6]], label %[[ENTRY_LS3:.*]], label %[[LS_FINAL_EXIT:.*]]
; CHECK: [[ENTRY_LS3]]:
; CHECK-NEXT: br label %[[LOOP_LS3:.*]]
; CHECK: [[LOOP_LS3]]:
-; CHECK-NEXT: [[I_LS3:%.*]] = phi i64 [ 300, %[[ENTRY_LS3]] ], [ [[I_NEXT_LS3:%.*]], %[[LOOP_LS3]] ]
+; CHECK-NEXT: [[I_LS3:%.*]] = phi i64 [ [[I_NEXT12]], %[[ENTRY_LS3]] ], [ [[I_NEXT_LS3:%.*]], %[[LOOP_LS3]] ]
; CHECK-NEXT: [[P_LS3:%.*]] = getelementptr i64, ptr [[A]], i64 [[I_LS3]]
; CHECK-NEXT: store i64 [[I_LS3]], ptr [[P_LS3]], align 4
; CHECK-NEXT: [[I_NEXT_LS3]] = add i64 [[I_LS3]], 1
diff --git a/llvm/test/Transforms/LoopSplit/multiple-partitions.ll b/llvm/test/Transforms/LoopSplit/multiple-partitions.ll
index 74346767d7edd..3ab9f646ec4f3 100644
--- a/llvm/test/Transforms/LoopSplit/multiple-partitions.ll
+++ b/llvm/test/Transforms/LoopSplit/multiple-partitions.ll
@@ -7,7 +7,7 @@
define void @three_way(ptr %a, i64 %n) {
; CHECK-LABEL: define void @three_way(
; CHECK-SAME: ptr [[A:%.*]], i64 [[N:%.*]]) {
-; CHECK-NEXT: [[LS_GUARD0:.*:]]
+; CHECK-NEXT: [[LS_GUARD0:.*]]:
; CHECK-NEXT: [[SMAX:%.*]] = call i64 @llvm.smax.i64(i64 [[N]], i64 1)
; CHECK-NEXT: [[TMP0:%.*]] = add nsw i64 [[SMAX]], -1
; CHECK-NEXT: [[SMIN:%.*]] = call i64 @llvm.smin.i64(i64 [[TMP0]], i64 49)
@@ -26,12 +26,13 @@ define void @three_way(ptr %a, i64 %n) {
; CHECK: [[EXIT]]:
; CHECK-NEXT: br label %[[LS_GUARD1]]
; CHECK: [[LS_GUARD1]]:
+; CHECK-NEXT: [[I_NEXT7:%.*]] = phi i64 [ [[I_NEXT]], %[[EXIT]] ], [ 0, %[[LS_GUARD0]] ]
; CHECK-NEXT: [[ITR_CHK3:%.*]] = icmp sle i64 50, [[SMIN1]]
; CHECK-NEXT: br i1 [[ITR_CHK3]], label %[[ENTRY_LS1:.*]], label %[[LS_GUARD2:.*]]
; CHECK: [[ENTRY_LS1]]:
; CHECK-NEXT: br label %[[LOOP_LS1:.*]]
; CHECK: [[LOOP_LS1]]:
-; CHECK-NEXT: [[I_LS1:%.*]] = phi i64 [ 50, %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
+; CHECK-NEXT: [[I_LS1:%.*]] = phi i64 [ [[I_NEXT7]], %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
; CHECK-NEXT: [[P_LS1:%.*]] = getelementptr i64, ptr [[A]], i64 [[I_LS1]]
; CHECK-NEXT: store i64 [[I_LS1]], ptr [[P_LS1]], align 4
; CHECK-NEXT: [[I_NEXT_LS1]] = add i64 [[I_LS1]], 1
@@ -40,12 +41,13 @@ define void @three_way(ptr %a, i64 %n) {
; CHECK: [[LS_EXIT1]]:
; CHECK-NEXT: br label %[[LS_GUARD2]]
; CHECK: [[LS_GUARD2]]:
+; CHECK-NEXT: [[I_NEXT8:%.*]] = phi i64 [ [[I_NEXT_LS1]], %[[LS_EXIT1]] ], [ [[I_NEXT7]], %[[LS_GUARD1]] ]
; CHECK-NEXT: [[ITR_CHK5:%.*]] = icmp sle i64 100, [[TMP0]]
; CHECK-NEXT: br i1 [[ITR_CHK5]], label %[[ENTRY_LS2:.*]], label %[[LS_FINAL_EXIT:.*]]
; CHECK: [[ENTRY_LS2]]:
; CHECK-NEXT: br label %[[LOOP_LS2:.*]]
; CHECK: [[LOOP_LS2]]:
-; CHECK-NEXT: [[I_LS2:%.*]] = phi i64 [ 100, %[[ENTRY_LS2]] ], [ [[I_NEXT_LS2:%.*]], %[[LOOP_LS2]] ]
+; CHECK-NEXT: [[I_LS2:%.*]] = phi i64 [ [[I_NEXT8]], %[[ENTRY_LS2]] ], [ [[I_NEXT_LS2:%.*]], %[[LOOP_LS2]] ]
; CHECK-NEXT: [[P_LS2:%.*]] = getelementptr i64, ptr [[A]], i64 [[I_LS2]]
; CHECK-NEXT: store i64 [[I_LS2]], ptr [[P_LS2]], align 4
; CHECK-NEXT: [[I_NEXT_LS2]] = add i64 [[I_LS2]], 1
diff --git a/llvm/test/Transforms/LoopSplit/reduction.ll b/llvm/test/Transforms/LoopSplit/reduction.ll
index 5b7d2ac2eae52..2af3ec66a1927 100644
--- a/llvm/test/Transforms/LoopSplit/reduction.ll
+++ b/llvm/test/Transforms/LoopSplit/reduction.ll
@@ -29,14 +29,15 @@ define i64 @reduction(ptr %a, i64 %n) {
; CHECK: [[EXIT]]:
; CHECK-NEXT: br label %[[LS_GUARD1]]
; CHECK: [[LS_GUARD1]]:
-; CHECK-NEXT: [[SUM_NEXT5:%.*]] = phi i64 [ [[SUM_NEXT]], %[[EXIT]] ], [ 0, %[[LS_GUARD0]] ]
+; CHECK-NEXT: [[SUM_NEXT6:%.*]] = phi i64 [ [[SUM_NEXT]], %[[EXIT]] ], [ 0, %[[LS_GUARD0]] ]
+; CHECK-NEXT: [[I_NEXT4:%.*]] = phi i64 [ [[I_NEXT]], %[[EXIT]] ], [ 0, %[[LS_GUARD0]] ]
; CHECK-NEXT: [[ITR_CHK2:%.*]] = icmp sle i64 50, [[TMP0]]
; CHECK-NEXT: br i1 [[ITR_CHK2]], label %[[ENTRY_LS1:.*]], label %[[LS_FINAL_EXIT:.*]]
; CHECK: [[ENTRY_LS1]]:
; CHECK-NEXT: br label %[[LOOP_LS1:.*]]
; CHECK: [[LOOP_LS1]]:
-; CHECK-NEXT: [[I_LS1:%.*]] = phi i64 [ 50, %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
-; CHECK-NEXT: [[SUM_LS1:%.*]] = phi i64 [ [[SUM_NEXT5]], %[[ENTRY_LS1]] ], [ [[SUM_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
+; CHECK-NEXT: [[I_LS1:%.*]] = phi i64 [ [[I_NEXT4]], %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
+; CHECK-NEXT: [[SUM_LS1:%.*]] = phi i64 [ [[SUM_NEXT6]], %[[ENTRY_LS1]] ], [ [[SUM_NEXT_LS1:%.*]], %[[LOOP_LS1]] ]
; CHECK-NEXT: [[P_LS1:%.*]] = getelementptr i64, ptr [[A]], i64 [[I_LS1]]
; CHECK-NEXT: [[V_LS1:%.*]] = load i64, ptr [[P_LS1]], align 4
; CHECK-NEXT: [[SUM_NEXT_LS1]] = add i64 [[SUM_LS1]], [[V_LS1]]
@@ -46,7 +47,7 @@ define i64 @reduction(ptr %a, i64 %n) {
; CHECK: [[LS_EXIT1]]:
; CHECK-NEXT: br label %[[LS_FINAL_EXIT]]
; CHECK: [[LS_FINAL_EXIT]]:
-; CHECK-NEXT: [[SUM_NEXT4:%.*]] = phi i64 [ [[SUM_NEXT_LS1]], %[[LS_EXIT1]] ], [ [[SUM_NEXT5]], %[[LS_GUARD1]] ]
+; CHECK-NEXT: [[SUM_NEXT4:%.*]] = phi i64 [ [[SUM_NEXT_LS1]], %[[LS_EXIT1]] ], [ [[SUM_NEXT6]], %[[LS_GUARD1]] ]
; CHECK-NEXT: ret i64 [[SUM_NEXT4]]
;
; The second partition's entry guard joins the first partition's result with the
>From 7289762e07f66f4953fc8ba07734831f220975a4 Mon Sep 17 00:00:00 2001
From: Ashutosh Nema <ashu1212 at gmail.com>
Date: Mon, 13 Jul 2026 16:26:03 +0530
Subject: [PATCH 3/6] [Transforms][Utils] LoopSplitUtils: support loops with
extra side exits
Accept a counted loop with additional "side" exits beyond the counted induction
exit. Only the counted exit is threaded through the guard chain; each side exit
stays a shared block that every partition's clone branches to. The counted exit
is peeled onto a dedicated block when shared, and a nested loop whose side exit
re-enters the enclosing loop is declined (irreducible flow). Side-exit LCSSA
PHIs gain one incoming per partition, and the trip count is taken from the
counted exit's own exact ExitCount for multi-exit loops.
Adds multi-exit.ll and multi-exit-reduction.ll.
---
llvm/lib/Transforms/Utils/LoopSplitUtils.cpp | 111 ++++++++++++++++--
.../LoopSplit/multi-exit-reduction.ll | 87 ++++++++++++++
llvm/test/Transforms/LoopSplit/multi-exit.ll | 76 ++++++++++++
3 files changed, 263 insertions(+), 11 deletions(-)
create mode 100644 llvm/test/Transforms/LoopSplit/multi-exit-reduction.ll
create mode 100644 llvm/test/Transforms/LoopSplit/multi-exit.ll
diff --git a/llvm/lib/Transforms/Utils/LoopSplitUtils.cpp b/llvm/lib/Transforms/Utils/LoopSplitUtils.cpp
index fb4b4b8858d55..283155363f6d5 100644
--- a/llvm/lib/Transforms/Utils/LoopSplitUtils.cpp
+++ b/llvm/lib/Transforms/Utils/LoopSplitUtils.cpp
@@ -33,12 +33,16 @@
// A descending (step -1) loop uses the same structure mirrored: partitions run
// high-to-low and the empty test, clamp, and predicates flip (>=/>).
//
+// Multi-exit loops: only the counted exit is threaded through the guard chain;
+// any "side" exit stays a shared block each partition's clone branches to, so
+// taking it leaves the whole chain. Its LCSSA PHIs gain one incoming/partition.
+//
// Usage guidelines:
// - Caller bounds must not wrap the induction type. The clamp absorbs a bound
// past the runtime trip count, but a Start +/- offset that overshoots the
// type extreme wraps in the bound arithmetic and cannot be repaired here.
-// - Bounds must be loop-invariant: they are expanded in guard0 (the preheader),
-// so a bound depending on a value defined inside the loop cannot be placed.
+// - Bounds must be loop-invariant: expanded in guard0 (the preheader), so a
+// bound depending on a value defined inside the loop cannot be placed.
// - The partitions must tile the original iteration space exactly -- same
// iterations, same order -- so the split preserves program behaviour.
// - A caller that drops a guard via avoidPartitionGuard() must itself ensure
@@ -48,6 +52,7 @@
#include "llvm/Transforms/Utils/LoopSplitUtils.h"
#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/Analysis/IVDescriptors.h"
#include "llvm/Analysis/LoopInfo.h"
#include "llvm/Analysis/ScalarEvolution.h"
@@ -104,6 +109,12 @@ struct LoopSplitUtils::SplitState {
/// Values that must survive across partitions (carried and/or live-out).
SmallVector<EscapingValue, 8> Escaping;
+ /// Exit edges of the loop other than the counted (induction) exit, as
+ /// (exiting block, exit block) pairs. Each is cloned per partition.
+ SmallVector<std::pair<BasicBlock *, BasicBlock *>, 4> SideExitEdges;
+ /// The distinct target blocks of \c SideExitEdges.
+ SmallPtrSet<BasicBlock *, 4> SideExitBlocks;
+
EscapingValue &addEscaping(Value *Def) {
Escaping.emplace_back();
Escaping.back().Def = Def;
@@ -241,16 +252,13 @@ bool LoopSplitUtils::isLegal() {
LLVM_DEBUG(dbgs() << "LS: missing preheader/latch\n");
return false;
}
- if (!L->getExitingBlock() || !L->getExitBlock()) {
- LLVM_DEBUG(dbgs() << "LS: not a single-exit loop\n");
- return false;
- }
if (!L->isLCSSAForm(*DT)) {
LLVM_DEBUG(dbgs() << "LS: loop is not in LCSSA form\n");
return false;
}
- // Identify the counted (induction) exit.
+ // Identify the counted (induction) exit. A loop with extra "side" exits is
+ // accepted -- they are cloned per partition in split().
const SCEVAddRecExpr *IndAR = analyzeInduction();
if (!IndAR) {
LLVM_DEBUG(dbgs() << "LS: no counted induction exit\n");
@@ -260,8 +268,11 @@ bool LoopSplitUtils::isLegal() {
if (!computeSignedness(IndAR))
return false;
- // A computable backedge-taken count fixes the iteration space we rebuild.
- const SCEV *BTC = SE->getBackedgeTakenCount(L);
+ // The trip count fixes the iteration space; it must be exact since split()
+ // replaces the counted test with the clamped bound. Single-exit uses the
+ // whole-loop count; multi-exit uses the counted exit's own exact count.
+ const SCEV *BTC = L->getExitingBlock() ? SE->getBackedgeTakenCount(L)
+ : SE->getExitCount(L, ExitingBlock);
if (isa<SCEVCouldNotCompute>(BTC)) {
LLVM_DEBUG(dbgs() << "LS: no computable trip count\n");
return false;
@@ -326,7 +337,60 @@ bool LoopSplitUtils::split() {
SplitState S;
S.OrigPreheader = L->getLoopPreheader();
- S.ExitBlock = L->getExitBlock();
+
+ // The counted exit is the unique out-of-loop successor of the counted exiting
+ // block; recompute it here since formDedicatedExitBlocks may have split it.
+ for (BasicBlock *Succ : successors(ExitingBlock))
+ if (!L->contains(Succ)) {
+ if (S.ExitBlock) // counted exiting block leaves the loop on two edges
+ return false;
+ S.ExitBlock = Succ;
+ }
+ if (!S.ExitBlock)
+ return false;
+
+ // A nested loop whose side exit re-enters the enclosing loop cannot be split:
+ // each clone adds another edge into it, giving irreducible flow. Decline
+ // before mutating IR. The counted exit is exempt (threaded linearly).
+ if (Loop *Parent = L->getParentLoop()) {
+ for (BasicBlock *BB : L->blocks())
+ for (BasicBlock *Succ : successors(BB))
+ if (!L->contains(Succ) &&
+ !(BB == ExitingBlock && Succ == S.ExitBlock) &&
+ Parent->contains(Succ))
+ return false;
+ }
+
+ // Isolating the counted-exit live-outs needs the exit reached solely from the
+ // counted exiting block. If the target is shared, peel the counted edge onto
+ // its own block (SplitBlockPredecessors fixes the LCSSA PHIs accordingly).
+ bool CountedExitShared = false;
+ for (BasicBlock *Pred : predecessors(S.ExitBlock))
+ if (Pred != ExitingBlock) {
+ CountedExitShared = true;
+ break;
+ }
+ if (CountedExitShared) {
+ // Cannot peel an EH pad; those are handled conservatively upstream.
+ if (S.ExitBlock->isEHPad())
+ return false;
+ BasicBlock *Dedicated =
+ SplitBlockPredecessors(S.ExitBlock, {ExitingBlock}, ".ls.counted.exit",
+ DT, LI, /*MSSAU=*/nullptr, /*PreserveLCSSA=*/true);
+ if (!Dedicated)
+ return false;
+ S.ExitBlock = Dedicated;
+ }
+
+ // Every other exit edge is a "side" exit: cloned per partition, its target's
+ // LCSSA PHIs gaining one incoming per partition (see clonePartitions()).
+ for (BasicBlock *BB : L->blocks())
+ for (BasicBlock *Succ : successors(BB))
+ if (!L->contains(Succ) && !(BB == ExitingBlock && Succ == S.ExitBlock)) {
+ S.SideExitEdges.emplace_back(BB, Succ);
+ S.SideExitBlocks.insert(Succ);
+ }
+
S.OuterLoop = LI->getLoopFor(S.ExitBlock);
collectEscapingValues(S);
@@ -507,6 +571,19 @@ void LoopSplitUtils::clonePartitions(SplitState &S) {
if (EV.CarriedHeaderPHI)
EV.PerPartitionPHI[I] = cast<PHINode>(VMap[EV.CarriedHeaderPHI]);
}
+
+ // Wire this partition's side-exit edges. The clone's side exiting block
+ // already branches to the shared side-exit block, so it is a new
+ // predecessor; add the matching LCSSA incoming (the cloned value).
+ for (auto &Edge : S.SideExitEdges) {
+ BasicBlock *OrigExiting = Edge.first;
+ BasicBlock *SideExit = Edge.second;
+ auto *ClonedExiting = cast<BasicBlock>(remapValue(VMap, OrigExiting));
+ for (PHINode &PN : SideExit->phis()) {
+ Value *Incoming = PN.getIncomingValueForBlock(OrigExiting);
+ PN.addIncoming(remapValue(VMap, Incoming), ClonedExiting);
+ }
+ }
}
}
@@ -605,6 +682,11 @@ void LoopSplitUtils::chainPartitions(SplitState &S) {
}
// The final exit is the last partition's merge target.
DT->changeImmediateDominator(S.FinalExit, MergeTargetIDom(Partitions.back()));
+
+ // Every side-exit block now has a predecessor in each partition; their common
+ // dominator is the entry guard, which dominates the whole chain.
+ for (BasicBlock *SideExit : S.SideExitBlocks)
+ DT->changeImmediateDominator(SideExit, S.EntryGuard);
}
// Rebuild SSA for every escaping value, repairing outside uses and seeding each
@@ -630,8 +712,15 @@ void LoopSplitUtils::reconstructSSA(SplitState &S) {
SmallVector<Use *, 8> OutsideUses;
for (Use &U : EV.Def->uses())
if (auto *User = dyn_cast<Instruction>(U.getUser()))
- if (!L->contains(User))
+ if (!L->contains(User)) {
+ // Side-exit LCSSA PHIs are repaired per-partition in
+ // clonePartitions(); don't rewrite them here (they need the running
+ // per-partition def, not the SSAUpdater's end-of-partition merge).
+ if (auto *P = dyn_cast<PHINode>(User))
+ if (S.SideExitBlocks.count(P->getParent()))
+ continue;
OutsideUses.push_back(&U);
+ }
for (Use *U : OutsideUses)
Updater.RewriteUse(*U);
}
diff --git a/llvm/test/Transforms/LoopSplit/multi-exit-reduction.ll b/llvm/test/Transforms/LoopSplit/multi-exit-reduction.ll
new file mode 100644
index 0000000000000..086734106c5aa
--- /dev/null
+++ b/llvm/test/Transforms/LoopSplit/multi-exit-reduction.ll
@@ -0,0 +1,87 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt -passes='loop-simplify,lcssa,loop-split-test,verify' \
+; RUN: -loop-split-points=2 -S < %s | FileCheck %s
+
+; A reduction that escapes through BOTH the side exit (early return of the
+; partial sum) and the counted exit. Each partition's clone feeds the shared
+; side-exit block's LCSSA PHI with its own running accumulator, which is seeded
+; from the preceding partition's join value; the counted-exit live-out is merged
+; separately at the final exit.
+
+define i32 @multi_exit_reduction(ptr %a, i32 %n, i32 %len) {
+; CHECK-LABEL: define i32 @multi_exit_reduction(
+; CHECK-SAME: ptr [[A:%.*]], i32 [[N:%.*]], i32 [[LEN:%.*]]) {
+; CHECK-NEXT: [[LS_GUARD0:.*]]:
+; CHECK-NEXT: [[SMAX:%.*]] = call i32 @llvm.smax.i32(i32 [[N]], i32 1)
+; CHECK-NEXT: [[TMP0:%.*]] = add nsw i32 [[SMAX]], -1
+; CHECK-NEXT: [[SMIN:%.*]] = call i32 @llvm.smin.i32(i32 [[TMP0]], i32 1)
+; CHECK-NEXT: [[ITR_CHK:%.*]] = icmp sle i32 0, [[SMIN]]
+; CHECK-NEXT: br i1 [[ITR_CHK]], label %[[ENTRY:.*]], label %[[LS_GUARD1:.*]]
+; CHECK: [[ENTRY]]:
+; CHECK-NEXT: br label %[[LOOP:.*]]
+; CHECK: [[LOOP]]:
+; CHECK-NEXT: [[I:%.*]] = phi i32 [ 0, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[CONT:.*]] ]
+; CHECK-NEXT: [[S:%.*]] = phi i32 [ 0, %[[ENTRY]] ], [ [[S_NEXT:%.*]], %[[CONT]] ]
+; CHECK-NEXT: [[OOB:%.*]] = icmp sge i32 [[I]], [[LEN]]
+; CHECK-NEXT: br i1 [[OOB]], label %[[EARLY:.*]], label %[[CONT]]
+; CHECK: [[CONT]]:
+; CHECK-NEXT: [[P:%.*]] = getelementptr i32, ptr [[A]], i32 [[I]]
+; CHECK-NEXT: [[V:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[S_NEXT]] = add i32 [[S]], [[V]]
+; CHECK-NEXT: [[I_NEXT]] = add nsw i32 [[I]], 1
+; CHECK-NEXT: [[ITR_CHK3:%.*]] = icmp sle i32 [[I_NEXT]], [[SMIN]]
+; CHECK-NEXT: br i1 [[ITR_CHK3]], label %[[LOOP]], label %[[EXIT:.*]]
+; CHECK: [[EARLY]]:
+; CHECK-NEXT: [[S_LCSSA1:%.*]] = phi i32 [ [[S]], %[[LOOP]] ], [ [[S_LS1:%.*]], %[[LOOP_LS1:.*]] ]
+; CHECK-NEXT: ret i32 [[S_LCSSA1]]
+; CHECK: [[EXIT]]:
+; CHECK-NEXT: br label %[[LS_GUARD1]]
+; CHECK: [[LS_GUARD1]]:
+; CHECK-NEXT: [[S_NEXT8:%.*]] = phi i32 [ [[S_NEXT]], %[[EXIT]] ], [ 0, %[[LS_GUARD0]] ]
+; CHECK-NEXT: [[I_NEXT6:%.*]] = phi i32 [ [[I_NEXT]], %[[EXIT]] ], [ 0, %[[LS_GUARD0]] ]
+; CHECK-NEXT: [[ITR_CHK4:%.*]] = icmp sle i32 2, [[TMP0]]
+; CHECK-NEXT: br i1 [[ITR_CHK4]], label %[[ENTRY_LS1:.*]], label %[[LS_FINAL_EXIT:.*]]
+; CHECK: [[ENTRY_LS1]]:
+; CHECK-NEXT: br label %[[LOOP_LS1]]
+; CHECK: [[LOOP_LS1]]:
+; CHECK-NEXT: [[I_LS1:%.*]] = phi i32 [ [[I_NEXT6]], %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[CONT_LS1:.*]] ]
+; CHECK-NEXT: [[S_LS1]] = phi i32 [ [[S_NEXT8]], %[[ENTRY_LS1]] ], [ [[S_NEXT_LS1:%.*]], %[[CONT_LS1]] ]
+; CHECK-NEXT: [[OOB_LS1:%.*]] = icmp sge i32 [[I_LS1]], [[LEN]]
+; CHECK-NEXT: br i1 [[OOB_LS1]], label %[[EARLY]], label %[[CONT_LS1]]
+; CHECK: [[CONT_LS1]]:
+; CHECK-NEXT: [[P_LS1:%.*]] = getelementptr i32, ptr [[A]], i32 [[I_LS1]]
+; CHECK-NEXT: [[V_LS1:%.*]] = load i32, ptr [[P_LS1]], align 4
+; CHECK-NEXT: [[S_NEXT_LS1]] = add i32 [[S_LS1]], [[V_LS1]]
+; CHECK-NEXT: [[I_NEXT_LS1]] = add nsw i32 [[I_LS1]], 1
+; CHECK-NEXT: [[ITR_CHK5:%.*]] = icmp sle i32 [[I_NEXT_LS1]], [[TMP0]]
+; CHECK-NEXT: br i1 [[ITR_CHK5]], label %[[LOOP_LS1]], label %[[LS_EXIT1:.*]]
+; CHECK: [[LS_EXIT1]]:
+; CHECK-NEXT: br label %[[LS_FINAL_EXIT]]
+; CHECK: [[LS_FINAL_EXIT]]:
+; CHECK-NEXT: [[S_NEXT6:%.*]] = phi i32 [ [[S_NEXT_LS1]], %[[LS_EXIT1]] ], [ [[S_NEXT8]], %[[LS_GUARD1]] ]
+; CHECK-NEXT: ret i32 [[S_NEXT6]]
+;
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %cont ]
+ %s = phi i32 [ 0, %entry ], [ %s.next, %cont ]
+ %oob = icmp sge i32 %i, %len
+ br i1 %oob, label %early, label %cont
+
+cont:
+ %p = getelementptr i32, ptr %a, i32 %i
+ %v = load i32, ptr %p
+ %s.next = add i32 %s, %v
+ %i.next = add nsw i32 %i, 1
+ %lc = icmp slt i32 %i.next, %n
+ br i1 %lc, label %loop, label %exit
+
+early:
+ ret i32 %s
+
+exit:
+ %s.lcssa = phi i32 [ %s.next, %cont ]
+ ret i32 %s.lcssa
+}
diff --git a/llvm/test/Transforms/LoopSplit/multi-exit.ll b/llvm/test/Transforms/LoopSplit/multi-exit.ll
new file mode 100644
index 0000000000000..75a2ef2b5db9f
--- /dev/null
+++ b/llvm/test/Transforms/LoopSplit/multi-exit.ll
@@ -0,0 +1,76 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt -passes='loop-simplify,lcssa,loop-split-test,verify' \
+; RUN: -loop-split-points=2 -S < %s | FileCheck %s
+
+; A loop with a side exit (the range-check branch to %oob) in addition to its
+; counted latch exit. The counted exit drives the guard chain; the side exit is
+; cloned into each partition and stays a shared block that every partition's
+; header branches to. The side-exit block has no live-out here.
+
+define void @multi_exit(ptr %arr, i32 %n, i32 %len) {
+; CHECK-LABEL: define void @multi_exit(
+; CHECK-SAME: ptr [[ARR:%.*]], i32 [[N:%.*]], i32 [[LEN:%.*]]) {
+; CHECK-NEXT: [[LS_GUARD0:.*]]:
+; CHECK-NEXT: [[SMAX:%.*]] = call i32 @llvm.smax.i32(i32 [[N]], i32 1)
+; CHECK-NEXT: [[TMP0:%.*]] = add nsw i32 [[SMAX]], -1
+; CHECK-NEXT: [[SMIN:%.*]] = call i32 @llvm.smin.i32(i32 [[TMP0]], i32 1)
+; CHECK-NEXT: [[ITR_CHK:%.*]] = icmp sle i32 0, [[SMIN]]
+; CHECK-NEXT: br i1 [[ITR_CHK]], label %[[ENTRY:.*]], label %[[LS_GUARD1:.*]]
+; CHECK: [[ENTRY]]:
+; CHECK-NEXT: br label %[[LOOP:.*]]
+; CHECK: [[LOOP]]:
+; CHECK-NEXT: [[I:%.*]] = phi i32 [ 0, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[BODY:.*]] ]
+; CHECK-NEXT: [[OOB:%.*]] = icmp sge i32 [[I]], [[LEN]]
+; CHECK-NEXT: br i1 [[OOB]], label %[[OUT:.*]], label %[[BODY]]
+; CHECK: [[BODY]]:
+; CHECK-NEXT: [[P:%.*]] = getelementptr i32, ptr [[ARR]], i32 [[I]]
+; CHECK-NEXT: store i32 0, ptr [[P]], align 4
+; CHECK-NEXT: [[I_NEXT]] = add nsw i32 [[I]], 1
+; CHECK-NEXT: [[ITR_CHK1:%.*]] = icmp sle i32 [[I_NEXT]], [[SMIN]]
+; CHECK-NEXT: br i1 [[ITR_CHK1]], label %[[LOOP]], label %[[EXIT:.*]]
+; CHECK: [[OUT]]:
+; CHECK-NEXT: ret void
+; CHECK: [[EXIT]]:
+; CHECK-NEXT: br label %[[LS_GUARD1]]
+; CHECK: [[LS_GUARD1]]:
+; CHECK-NEXT: [[I_NEXT4:%.*]] = phi i32 [ [[I_NEXT]], %[[EXIT]] ], [ 0, %[[LS_GUARD0]] ]
+; CHECK-NEXT: [[ITR_CHK2:%.*]] = icmp sle i32 2, [[TMP0]]
+; CHECK-NEXT: br i1 [[ITR_CHK2]], label %[[ENTRY_LS1:.*]], label %[[LS_FINAL_EXIT:.*]]
+; CHECK: [[ENTRY_LS1]]:
+; CHECK-NEXT: br label %[[LOOP_LS1:.*]]
+; CHECK: [[LOOP_LS1]]:
+; CHECK-NEXT: [[I_LS1:%.*]] = phi i32 [ [[I_NEXT4]], %[[ENTRY_LS1]] ], [ [[I_NEXT_LS1:%.*]], %[[BODY_LS1:.*]] ]
+; CHECK-NEXT: [[OOB_LS1:%.*]] = icmp sge i32 [[I_LS1]], [[LEN]]
+; CHECK-NEXT: br i1 [[OOB_LS1]], label %[[OUT]], label %[[BODY_LS1]]
+; CHECK: [[BODY_LS1]]:
+; CHECK-NEXT: [[P_LS1:%.*]] = getelementptr i32, ptr [[ARR]], i32 [[I_LS1]]
+; CHECK-NEXT: store i32 0, ptr [[P_LS1]], align 4
+; CHECK-NEXT: [[I_NEXT_LS1]] = add nsw i32 [[I_LS1]], 1
+; CHECK-NEXT: [[ITR_CHK3:%.*]] = icmp sle i32 [[I_NEXT_LS1]], [[TMP0]]
+; CHECK-NEXT: br i1 [[ITR_CHK3]], label %[[LOOP_LS1]], label %[[LS_EXIT1:.*]]
+; CHECK: [[LS_EXIT1]]:
+; CHECK-NEXT: br label %[[LS_FINAL_EXIT]]
+; CHECK: [[LS_FINAL_EXIT]]:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %body ]
+ %oob = icmp sge i32 %i, %len
+ br i1 %oob, label %out, label %body
+
+body:
+ %p = getelementptr i32, ptr %arr, i32 %i
+ store i32 0, ptr %p
+ %i.next = add nsw i32 %i, 1
+ %lc = icmp slt i32 %i.next, %n
+ br i1 %lc, label %loop, label %exit
+
+out:
+ ret void
+
+exit:
+ ret void
+}
>From cae86e3394fdba0821799a32cf83ff0d10e46589 Mon Sep 17 00:00:00 2001
From: Ashutosh Nema <ashu1212 at gmail.com>
Date: Mon, 13 Jul 2026 16:27:48 +0530
Subject: [PATCH 4/6] [Transforms][Utils] LoopSplitUtils:
uncomputable-trip-count and narrow-latch fallbacks
Add two opt-in relaxations, both off by default. AllowUncomputableTripCount
splits a loop whose counted exit has no computable trip count: interior
partitions are clamped against the invariant bound and the original latch, and
the final partition keeps the original counted latch. AllowTruncatedLatchCompare
accepts a counted exit that compares a truncation of the wider induction, with
the clamp emitted on the wide value.
Exposes isUncomputableTripCountMode()/getInductionBound() and a
-loop-split-allow-uncomputable-trip-count driver option. Adds
multi-exit-inexact-bound.ll and multi-exit-inexact-bound-declines.ll.
---
.../llvm/Transforms/Utils/LoopSplitUtils.h | 32 ++++-
.../Transforms/Utils/LoopSplitTestPass.cpp | 15 ++-
llvm/lib/Transforms/Utils/LoopSplitUtils.cpp | 124 +++++++++++++++++-
.../multi-exit-inexact-bound-declines.ll | 37 ++++++
.../LoopSplit/multi-exit-inexact-bound.ll | 81 ++++++++++++
5 files changed, 280 insertions(+), 9 deletions(-)
create mode 100644 llvm/test/Transforms/LoopSplit/multi-exit-inexact-bound-declines.ll
create mode 100644 llvm/test/Transforms/LoopSplit/multi-exit-inexact-bound.ll
diff --git a/llvm/include/llvm/Transforms/Utils/LoopSplitUtils.h b/llvm/include/llvm/Transforms/Utils/LoopSplitUtils.h
index e4878764b445f..fa3bf42aaf13d 100644
--- a/llvm/include/llvm/Transforms/Utils/LoopSplitUtils.h
+++ b/llvm/include/llvm/Transforms/Utils/LoopSplitUtils.h
@@ -47,8 +47,17 @@ class Value;
/// \endcode
class LoopSplitUtils {
public:
- LoopSplitUtils(Loop *L, LoopInfo *LI, ScalarEvolution *SE, DominatorTree *DT)
- : L(L), LI(LI), SE(SE), DT(DT) {}
+ /// \p AllowUncomputableTripCount: also split a multi-exit loop whose counted
+ /// exit has no computable trip count (final partition keeps the original
+ /// latch). Default off.
+ /// \p AllowTruncatedLatchCompare: also split when the counted exit compares a
+ /// truncation of the wider induction (which still drives partitioning). Off.
+ LoopSplitUtils(Loop *L, LoopInfo *LI, ScalarEvolution *SE, DominatorTree *DT,
+ bool AllowUncomputableTripCount = false,
+ bool AllowTruncatedLatchCompare = false)
+ : L(L), LI(LI), SE(SE), DT(DT),
+ AllowUncomputableTripCount(AllowUncomputableTripCount),
+ AllowTruncatedLatchCompare(AllowTruncatedLatchCompare) {}
/// Analyze \p L; true if it is a counted loop we can split: LCSSA, a counted
/// exit in the latch or header, a constant-step integer induction, and an
@@ -62,6 +71,14 @@ class LoopSplitUtils {
/// the space the partitions tile. Valid only after isLegal() succeeds.
const SCEV *getInductionEnd() const { return InductionEnd; }
+ /// True if isLegal() accepted the loop via the uncomputable-trip-count
+ /// fallback (no exact count; the original latch drives the final partition).
+ bool isUncomputableTripCountMode() const { return UncomputableTripCountMode; }
+
+ /// The counted exit's invariant bound, used as the final partition's end in
+ /// uncomputable-trip-count mode. Null unless isUncomputableTripCountMode().
+ const SCEV *getInductionBound() const { return InductionBound; }
+
/// Append an inclusive partition range [Start, End] in iteration order.
/// Partitions must tile the whole space: first Start = induction start, each
/// later Start = previous End +/- step, last End = induction end (desc: S >= E).
@@ -132,6 +149,8 @@ class LoopSplitUtils {
LoopInfo *LI;
ScalarEvolution *SE;
DominatorTree *DT;
+ bool AllowUncomputableTripCount = false; // opt-in to the fallback below.
+ bool AllowTruncatedLatchCompare = false; // opt-in to a trunc(iv) exit compare.
// Induction analysis, populated by isLegal().
PHINode *Induction = nullptr;
@@ -146,6 +165,15 @@ class LoopSplitUtils {
APInt InductionStep; // signed constant step, induction width.
const SCEV *InductionEnd = nullptr;
+ // Uncomputable-trip-count fallback state, set by isLegal() only when there is
+ // no exact count and AllowUncomputableTripCount is set; inert by default.
+ bool UncomputableTripCountMode = false; // accepted via the fallback path.
+ const SCEV *InductionBound =
+ nullptr; // counted compare's invariant bound SCEV.
+ Value *InductionBoundVal = nullptr; // that bound as an IR value (invariant).
+ unsigned CountedContinuePred =
+ 0; // ICmpInst pred P for "IndOp P Bound" == continue.
+
/// One record per partition, in add order.
SmallVector<PartitionInfo, 4> Partitions;
diff --git a/llvm/lib/Transforms/Utils/LoopSplitTestPass.cpp b/llvm/lib/Transforms/Utils/LoopSplitTestPass.cpp
index f650cac65192b..088fd5f28618a 100644
--- a/llvm/lib/Transforms/Utils/LoopSplitTestPass.cpp
+++ b/llvm/lib/Transforms/Utils/LoopSplitTestPass.cpp
@@ -47,11 +47,19 @@ static cl::opt<bool> PrintPartitionMap(
"counterpart in every partition (LoopSplitUtils::getPartitionValue)"),
cl::init(false));
+static cl::opt<bool> AllowUncomputableTripCount(
+ "loop-split-allow-uncomputable-trip-count",
+ cl::desc("Opt in to LoopSplitUtils' uncomputable-trip-count fallback, which "
+ "splits a multi-exit loop whose counted exit has no computable trip "
+ "count (e.g. an ascending non-unit-step loop with a symbolic bound) "
+ "by keeping the original latch on the final partition"),
+ cl::init(false));
+
/// Build the partition list for \p L from the command-line split offsets and
/// run the transform. Returns true if the loop was split.
static bool splitLoop(Loop *L, ScalarEvolution &SE, DominatorTree &DT,
LoopInfo &LI) {
- LoopSplitUtils LSU(L, &LI, &SE, &DT);
+ LoopSplitUtils LSU(L, &LI, &SE, &DT, AllowUncomputableTripCount);
if (!LSU.isLegal()) {
LLVM_DEBUG(dbgs() << "loop-split-test: loop is not legal for splitting\n");
return false;
@@ -64,7 +72,10 @@ static bool splitLoop(Loop *L, ScalarEvolution &SE, DominatorTree &DT,
const SCEV *Start = IndAR->getStart();
// Use the utility's counted induction end (works for multi-exit loops too).
- const SCEV *End = LSU.getInductionEnd();
+ // In the uncomputable-trip-count fallback there is no exact end, so drive the
+ // final partition with the invariant counted bound (the utility keeps latch).
+ const SCEV *End = LSU.isUncomputableTripCountMode() ? LSU.getInductionBound()
+ : LSU.getInductionEnd();
Type *Ty = Start->getType();
if (End->getType() != Ty)
End = SE.getTruncateExpr(End, Ty);
diff --git a/llvm/lib/Transforms/Utils/LoopSplitUtils.cpp b/llvm/lib/Transforms/Utils/LoopSplitUtils.cpp
index 283155363f6d5..c6b8d77fbde35 100644
--- a/llvm/lib/Transforms/Utils/LoopSplitUtils.cpp
+++ b/llvm/lib/Transforms/Utils/LoopSplitUtils.cpp
@@ -64,6 +64,7 @@
#include "llvm/IR/Function.h"
#include "llvm/IR/IRBuilder.h"
#include "llvm/IR/Instructions.h"
+#include "llvm/IR/PatternMatch.h"
#include "llvm/Support/Debug.h"
#include "llvm/Transforms/Utils/BasicBlockUtils.h"
#include "llvm/Transforms/Utils/Cloning.h"
@@ -73,6 +74,7 @@
#include "llvm/Transforms/Utils/ValueMapper.h"
using namespace llvm;
+using namespace llvm::PatternMatch;
#define DEBUG_TYPE "loop-split-utils"
@@ -170,8 +172,9 @@ const SCEVAddRecExpr *LoopSplitUtils::analyzeInduction() {
BasicBlock *Header = L->getHeader();
BasicBlock *Latch = L->getLoopLatch();
- // The counted exit lives in the latch (bottom-tested) or header (top-tested).
- // Try the latch first so a single-exit loop keeps its previous classification.
+ // The counted exit lives in the latch (bottom-tested) or header (top-tested);
+ // any other exiting block is tolerated as a side exit. Try the latch first so
+ // a single-exit loop keeps its previous classification.
SmallVector<BasicBlock *, 2> Candidates;
Candidates.push_back(Latch);
if (Header != Latch)
@@ -209,6 +212,17 @@ const SCEVAddRecExpr *LoopSplitUtils::analyzeInduction() {
UsesPHI = true;
else if (CmpOp0 == StepVal || CmpOp1 == StepVal)
UsesPHI = false;
+ else if (AllowTruncatedLatchCompare &&
+ (match(CmpOp0, m_Trunc(m_Specific(&PN))) ||
+ match(CmpOp1, m_Trunc(m_Specific(&PN)))))
+ // Narrow latch: the exit compares "trunc(PHI)". The wide PHI still
+ // drives the partitions; the clamp is emitted on the wide value.
+ UsesPHI = true;
+ else if (AllowTruncatedLatchCompare &&
+ (match(CmpOp0, m_Trunc(m_Specific(StepVal))) ||
+ match(CmpOp1, m_Trunc(m_Specific(StepVal)))))
+ // Narrow latch on the post-increment value: "trunc(PHI+step)".
+ UsesPHI = false;
else
continue;
@@ -274,10 +288,51 @@ bool LoopSplitUtils::isLegal() {
const SCEV *BTC = L->getExitingBlock() ? SE->getBackedgeTakenCount(L)
: SE->getExitCount(L, ExitingBlock);
if (isa<SCEVCouldNotCompute>(BTC)) {
- LLVM_DEBUG(dbgs() << "LS: no computable trip count\n");
- return false;
+ // Uncomputable-trip-count fallback (opt-in): with no exact count, drive the
+ // final partition with the original counted latch and clamp interior ones
+ // against the invariant bound (mirrors IRCE). Only for declined loops.
+ if (!AllowUncomputableTripCount) {
+ LLVM_DEBUG(dbgs() << "LS: no computable trip count\n");
+ return false;
+ }
+ // The counted compare's other operand is the loop-invariant bound; use the
+ // IR value directly (loop-invariance guarantees it dominates the guards).
+ Value *BoundVal = LatchCmp->getOperand(0) == LatchIndOperand
+ ? LatchCmp->getOperand(1)
+ : LatchCmp->getOperand(0);
+ if (!L->isLoopInvariant(BoundVal)) {
+ LLVM_DEBUG(dbgs() << "LS: counted bound is not loop-invariant\n");
+ return false;
+ }
+ if (BoundVal->getType() != IndAR->getStart()->getType()) {
+ LLVM_DEBUG(dbgs() << "LS: counted bound type mismatch\n");
+ return false;
+ }
+ // Normalize the predicate to read "IndOp <pred> Bound" meaning "continue",
+ // accounting for operand order and which branch successor stays in the loop.
+ ICmpInst::Predicate Raw =
+ LatchCmp->getOperand(0) == LatchIndOperand
+ ? LatchCmp->getPredicate()
+ : ICmpInst::getSwappedPredicate(LatchCmp->getPredicate());
+ auto *CountedBr = cast<BranchInst>(ExitingBlock->getTerminator());
+ bool ContinueWhenTrue = L->contains(CountedBr->getSuccessor(0));
+ ICmpInst::Predicate ContinuePred =
+ ContinueWhenTrue ? Raw : ICmpInst::getInversePredicate(Raw);
+
+ InductionBoundVal = BoundVal;
+ InductionBound = SE->getSCEV(BoundVal);
+ CountedContinuePred = static_cast<unsigned>(ContinuePred);
+ UncomputableTripCountMode = true;
+ return true;
}
+ // Narrow latch: the trip count is in the narrower latch type but the induction
+ // is wider. Zero-extend the count to the induction type (a trip count is a
+ // non-negative index, so the extend is exact) for evaluateAtIteration.
+ if (AllowTruncatedLatchCompare &&
+ BTC->getType() != IndAR->getStart()->getType())
+ BTC = SE->getNoopOrZeroExtend(BTC, IndAR->getStart()->getType());
+
// evaluateAtIteration(BTC) is the induction value at the final exit test: the
// last value the body ran with when bottom-tested; for top-tested the body
// ran a step earlier, so step back to the last executed value.
@@ -502,6 +557,14 @@ void LoopSplitUtils::expandPartitionBounds(SplitState &S) {
P.StartVal = Expander.expandCodeFor(P.StartExpr, IndTy, EntryGuardTerm);
+ // Uncomputable-trip-count mode: no exact end to clamp against. Interior
+ // partitions stop at their own constant end AND the original latch (in
+ // rewriteLatch); the final keeps it, so the clamp target is the part's end.
+ if (UncomputableTripCountMode) {
+ P.SelEnd = Expander.expandCodeFor(P.EndExpr, IndTy, EntryGuardTerm);
+ continue;
+ }
+
// Clamp the end to the induction end (min ascending, max descending) so a
// short trip count keeps the last iteration in the right partition.
const SCEV *ClampedEndSCEV;
@@ -600,6 +663,33 @@ void LoopSplitUtils::rewriteLatch(Loop *PL, BasicBlock *ExitingBlk,
: Term->getSuccessor(1);
IRBuilder<> B(Cmp);
+ // Uncomputable-trip-count mode: keep the loop's counted condition so it is
+ // driven by the original latch. The final partition uses it alone; an interior
+ // one ANDs it with the clamp to its boundary to hand off to the next first.
+ if (UncomputableTripCountMode) {
+ ICmpInst::Predicate OrigPred =
+ static_cast<ICmpInst::Predicate>(CountedContinuePred);
+ Value *OrigContinue =
+ B.CreateICmp(OrigPred, IndOp, InductionBoundVal, "cnt.chk");
+ Value *Continue = OrigContinue;
+ if (!IsLastPartition) {
+ Value *Bound = SelEnd;
+ if (Bound->getType() != IndOp->getType())
+ Bound = B.CreateIntCast(Bound, IndOp->getType(), InductionIsSigned);
+ bool Inclusive = LatchUsesInductionPHI == IsTopTested;
+ ICmpInst::Predicate ClampPred = continuePredicate(
+ InductionIsSigned, InductionIsDescending, Inclusive);
+ Value *ClampCmp = B.CreateICmp(ClampPred, IndOp, Bound, "itr.chk");
+ Continue = B.CreateAnd(ClampCmp, OrigContinue, "ls.cont");
+ }
+ B.SetInsertPoint(Term);
+ B.CreateCondBr(Continue, ContinueTarget, Exit);
+ Term->eraseFromParent();
+ if (Cmp->use_empty())
+ Cmp->eraseFromParent();
+ return;
+ }
+
Value *Bound = SelEnd;
if (Bound->getType() != IndOp->getType())
Bound = B.CreateIntCast(Bound, IndOp->getType(), InductionIsSigned);
@@ -623,6 +713,12 @@ void LoopSplitUtils::chainPartitions(SplitState &S) {
const ICmpInst::Predicate GuardPred =
guardPredicate(InductionIsSigned, InductionIsDescending);
+ // Uncomputable-trip-count mode has no exact end for partition 0's usual guard.
+ // A bottom-tested loop always runs its first iteration, so partition 0 enters
+ // unconditionally (top-tested is conditional, so its guard is kept).
+ if (UncomputableTripCountMode && !IsTopTested && getNumPartitions() > 0)
+ Partitions[0].Guarded = false;
+
// Emit each guard, clamp each latch, and chain partitions; a skipped
// partition falls through to the next guard.
const unsigned N = getNumPartitions();
@@ -655,7 +751,25 @@ void LoopSplitUtils::chainPartitions(SplitState &S) {
IRBuilder<>(GuardTerm).CreateBr(P.Preheader);
} else {
IRBuilder<> B(GuardTerm);
- Value *Enter = B.CreateICmp(GuardPred, P.StartVal, P.SelEnd, "itr.chk");
+ Value *Enter;
+ if (UncomputableTripCountMode) {
+ // No exact end: enter iff the original loop runs this partition's first
+ // iteration. Map the start to the value the counted compare inspects for
+ // that iteration, then apply the original predicate against the bound.
+ int DeltaSteps =
+ (LatchUsesInductionPHI ? 0 : 1) - (IsTopTested ? 0 : 1);
+ Value *GuardVal = P.StartVal;
+ if (DeltaSteps != 0) {
+ APInt Delta = InductionStep * DeltaSteps;
+ GuardVal = B.CreateAdd(
+ P.StartVal, ConstantInt::get(P.StartVal->getType(), Delta));
+ }
+ Enter =
+ B.CreateICmp(static_cast<ICmpInst::Predicate>(CountedContinuePred),
+ GuardVal, InductionBoundVal, "cnt.chk");
+ } else {
+ Enter = B.CreateICmp(GuardPred, P.StartVal, P.SelEnd, "itr.chk");
+ }
B.CreateCondBr(Enter, P.Preheader, MergeAfter);
}
GuardTerm->eraseFromParent();
diff --git a/llvm/test/Transforms/LoopSplit/multi-exit-inexact-bound-declines.ll b/llvm/test/Transforms/LoopSplit/multi-exit-inexact-bound-declines.ll
new file mode 100644
index 0000000000000..a5a6a81fbd435
--- /dev/null
+++ b/llvm/test/Transforms/LoopSplit/multi-exit-inexact-bound-declines.ll
@@ -0,0 +1,37 @@
+; RUN: opt -passes='loop-simplify,lcssa,loop-split-test,verify' \
+; RUN: -loop-split-points=2 -S < %s | FileCheck %s
+
+; Without -loop-split-allow-uncomputable-trip-count the fallback is off, so this
+; non-unit-step symbolic-bound multi-exit loop (no computable trip count) is
+; declined and left untouched -- the feature is strictly opt-in and additive.
+; (multi-exit-inexact-bound.ll checks the same loop *is* split with the flag.)
+
+; CHECK-LABEL: define i32 @multi_inexact(
+; CHECK-NOT: ls.guard
+; CHECK-NOT: itr.chk
+; CHECK-NOT: .ls1
+
+define i32 @multi_inexact(i32 %n) {
+entry:
+ br label %h
+
+h:
+ %iv = phi i32 [ 0, %entry ], [ %inc, %l ]
+ %acc = phi i32 [ 0, %entry ], [ %an, %l ]
+ %brk = icmp eq i32 %iv, 6
+ br i1 %brk, label %side, label %l
+
+l:
+ %an = add i32 %acc, %iv
+ %inc = add nuw nsw i32 %iv, 2
+ %ec = icmp slt i32 %inc, %n
+ br i1 %ec, label %h, label %exit
+
+exit:
+ %r = phi i32 [ %an, %l ]
+ ret i32 %r
+
+side:
+ %rs = phi i32 [ %acc, %h ]
+ ret i32 %rs
+}
diff --git a/llvm/test/Transforms/LoopSplit/multi-exit-inexact-bound.ll b/llvm/test/Transforms/LoopSplit/multi-exit-inexact-bound.ll
new file mode 100644
index 0000000000000..043dd290e4e56
--- /dev/null
+++ b/llvm/test/Transforms/LoopSplit/multi-exit-inexact-bound.ll
@@ -0,0 +1,81 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt -passes='loop-simplify,lcssa,loop-split-test,verify' \
+; RUN: -loop-split-points=2 -loop-split-allow-uncomputable-trip-count -S < %s \
+; RUN: | FileCheck %s
+
+; A multi-exit loop with a non-unit step (2) and a symbolic bound (%n): the
+; counted latch exit has no SCEV-computable trip count, so the default utility
+; declines it. With -loop-split-allow-uncomputable-trip-count the fallback splits
+; it, clamping the interior partition to its own constant end ANDed with the
+; original counted condition, and driving the final partition with the original
+; latch (self-terminating). The bottom-tested first partition enters
+; unconditionally, matching the original do-while semantics.
+
+define i32 @multi_inexact(i32 %n) {
+; CHECK-LABEL: define i32 @multi_inexact(
+; CHECK-SAME: i32 [[N:%.*]]) {
+; CHECK-NEXT: [[LS_GUARD0:.*:]]
+; CHECK-NEXT: br label %[[ENTRY:.*]]
+; CHECK: [[ENTRY]]:
+; CHECK-NEXT: br label %[[H:.*]]
+; CHECK: [[H]]:
+; CHECK-NEXT: [[IV:%.*]] = phi i32 [ 0, %[[ENTRY]] ], [ [[INC:%.*]], %[[L:.*]] ]
+; CHECK-NEXT: [[ACC:%.*]] = phi i32 [ 0, %[[ENTRY]] ], [ [[AN:%.*]], %[[L]] ]
+; CHECK-NEXT: [[BRK:%.*]] = icmp eq i32 [[IV]], 6
+; CHECK-NEXT: br i1 [[BRK]], label %[[SIDE:.*]], label %[[L]]
+; CHECK: [[L]]:
+; CHECK-NEXT: [[AN]] = add i32 [[ACC]], [[IV]]
+; CHECK-NEXT: [[INC]] = add nuw nsw i32 [[IV]], 2
+; CHECK-NEXT: [[CNT_CHK:%.*]] = icmp slt i32 [[INC]], [[N]]
+; CHECK-NEXT: [[ITR_CHK:%.*]] = icmp sle i32 [[INC]], 1
+; CHECK-NEXT: [[LS_CONT:%.*]] = and i1 [[ITR_CHK]], [[CNT_CHK]]
+; CHECK-NEXT: br i1 [[LS_CONT]], label %[[H]], label %[[EXIT:.*]]
+; CHECK: [[EXIT]]:
+; CHECK-NEXT: br label %[[LS_GUARD1:.*]]
+; CHECK: [[LS_GUARD1]]:
+; CHECK-NEXT: [[CNT_CHK1:%.*]] = icmp slt i32 2, [[N]]
+; CHECK-NEXT: br i1 [[CNT_CHK1]], label %[[ENTRY_LS1:.*]], label %[[LS_FINAL_EXIT:.*]]
+; CHECK: [[ENTRY_LS1]]:
+; CHECK-NEXT: br label %[[H_LS1:.*]]
+; CHECK: [[H_LS1]]:
+; CHECK-NEXT: [[IV_LS1:%.*]] = phi i32 [ [[INC]], %[[ENTRY_LS1]] ], [ [[INC_LS1:%.*]], %[[L_LS1:.*]] ]
+; CHECK-NEXT: [[ACC_LS1:%.*]] = phi i32 [ [[AN]], %[[ENTRY_LS1]] ], [ [[AN_LS1:%.*]], %[[L_LS1]] ]
+; CHECK-NEXT: [[BRK_LS1:%.*]] = icmp eq i32 [[IV_LS1]], 6
+; CHECK-NEXT: br i1 [[BRK_LS1]], label %[[SIDE]], label %[[L_LS1]]
+; CHECK: [[L_LS1]]:
+; CHECK-NEXT: [[AN_LS1]] = add i32 [[ACC_LS1]], [[IV_LS1]]
+; CHECK-NEXT: [[INC_LS1]] = add nuw nsw i32 [[IV_LS1]], 2
+; CHECK-NEXT: [[CNT_CHK2:%.*]] = icmp slt i32 [[INC_LS1]], [[N]]
+; CHECK-NEXT: br i1 [[CNT_CHK2]], label %[[H_LS1]], label %[[LS_EXIT1:.*]]
+; CHECK: [[LS_EXIT1]]:
+; CHECK-NEXT: br label %[[LS_FINAL_EXIT]]
+; CHECK: [[LS_FINAL_EXIT]]:
+; CHECK-NEXT: [[AN3:%.*]] = phi i32 [ [[AN_LS1]], %[[LS_EXIT1]] ], [ [[AN]], %[[LS_GUARD1]] ]
+; CHECK-NEXT: ret i32 [[AN3]]
+; CHECK: [[SIDE]]:
+; CHECK-NEXT: [[RS:%.*]] = phi i32 [ [[ACC]], %[[H]] ], [ [[ACC_LS1]], %[[H_LS1]] ]
+; CHECK-NEXT: ret i32 [[RS]]
+;
+entry:
+ br label %h
+
+h:
+ %iv = phi i32 [ 0, %entry ], [ %inc, %l ]
+ %acc = phi i32 [ 0, %entry ], [ %an, %l ]
+ %brk = icmp eq i32 %iv, 6
+ br i1 %brk, label %side, label %l
+
+l:
+ %an = add i32 %acc, %iv
+ %inc = add nuw nsw i32 %iv, 2
+ %ec = icmp slt i32 %inc, %n
+ br i1 %ec, label %h, label %exit
+
+exit:
+ %r = phi i32 [ %an, %l ]
+ ret i32 %r
+
+side:
+ %rs = phi i32 [ %acc, %h ]
+ ret i32 %rs
+}
>From 49cf7ce7b51b1708734755e852183402053964ce Mon Sep 17 00:00:00 2001
From: Ashutosh Nema <ashu1212 at gmail.com>
Date: Mon, 13 Jul 2026 16:29:58 +0530
Subject: [PATCH 5/6] [IRCE] Optionally restructure loops via LoopSplitUtils
(default off, NFC)
Add -irce-use-loop-split-utils (hidden, default off): when set, IRCE drives its
pre/main/post range restructuring through the generic LoopSplitUtils instead of
the bespoke LoopConstrainer, folding eliminated checks in the main partition.
Boundaries mirror LoopConstrainer::run for both directions, non-main partitions
get the same disable-loop-opts / clone-tag / sibling treatment, and nested
EH-pad side exits are declined. With the option off there is no behavioural
change.
Adds loop-split-utils-driver.ll and loop-split-utils-driver-narrow-latch.ll.
---
.../Scalar/InductiveRangeCheckElimination.cpp | 223 +++++++++++++++++-
.../loop-split-utils-driver-narrow-latch.ll | 39 +++
.../IRCE/loop-split-utils-driver.ll | 112 +++++++++
3 files changed, 372 insertions(+), 2 deletions(-)
create mode 100644 llvm/test/Transforms/IRCE/loop-split-utils-driver-narrow-latch.ll
create mode 100644 llvm/test/Transforms/IRCE/loop-split-utils-driver.ll
diff --git a/llvm/lib/Transforms/Scalar/InductiveRangeCheckElimination.cpp b/llvm/lib/Transforms/Scalar/InductiveRangeCheckElimination.cpp
index 98da1e9225172..d56cd093d34c1 100644
--- a/llvm/lib/Transforms/Scalar/InductiveRangeCheckElimination.cpp
+++ b/llvm/lib/Transforms/Scalar/InductiveRangeCheckElimination.cpp
@@ -67,6 +67,7 @@
#include "llvm/IR/Instructions.h"
#include "llvm/IR/Metadata.h"
#include "llvm/IR/Module.h"
+#include "llvm/IR/Operator.h"
#include "llvm/IR/PatternMatch.h"
#include "llvm/IR/Type.h"
#include "llvm/IR/Use.h"
@@ -83,6 +84,7 @@
#include "llvm/Transforms/Utils/Cloning.h"
#include "llvm/Transforms/Utils/LoopConstrainer.h"
#include "llvm/Transforms/Utils/LoopSimplify.h"
+#include "llvm/Transforms/Utils/LoopSplitUtils.h"
#include "llvm/Transforms/Utils/LoopUtils.h"
#include "llvm/Transforms/Utils/ValueMapper.h"
#include <algorithm>
@@ -126,6 +128,19 @@ static cl::opt<bool>
PrintScaledBoundaryRangeChecks("irce-print-scaled-boundary-range-checks",
cl::Hidden, cl::init(false));
+// When enabled, IRCE restructures the loop's pre/main/post ranges via the
+// generic LoopSplitUtils instead of the bespoke LoopConstrainer -- behaviorally
+// equivalent but structurally different IR. Default off (unchanged output).
+static cl::opt<bool> UseLoopSplitUtils(
+ "irce-use-loop-split-utils", cl::init(false), cl::Hidden,
+ cl::desc(
+ "Drive IRCE's loop restructuring through LoopSplitUtils instead of "
+ "LoopConstrainer where possible"));
+
+// Metadata tag LoopConstrainer stamps on its clones so IRCE skips reprocessing
+// them; the LoopSplitUtils driver reuses it on the pre/post partitions.
+static const char *LoopConstrainerClonedLoopTag = "loop_constrainer.loop.clone";
+
#define DEBUG_TYPE "irce"
namespace {
@@ -985,6 +1000,195 @@ InductiveRangeCheckElimination::estimatedTripCount(const Loop &L) {
return {ExitProbability.scaleByInverse(1)};
}
+// Mark \p L so no further loop optimization runs on it, mirroring
+// LoopConstrainer's DisableAllLoopOptsOnLoop for the pre/post partitions.
+static void disableAllLoopOptsOnLoop(Loop &L) {
+ LLVMContext &Context = L.getHeader()->getContext();
+ MDNode *Dummy = MDNode::get(Context, {});
+ MDNode *DisableUnroll = MDNode::get(
+ Context, {MDString::get(Context, "llvm.loop.unroll.disable")});
+ Metadata *FalseVal =
+ ConstantAsMetadata::get(ConstantInt::get(Type::getInt1Ty(Context), 0));
+ MDNode *DisableVectorize = MDNode::get(
+ Context,
+ {MDString::get(Context, "llvm.loop.vectorize.enable"), FalseVal});
+ MDNode *DisableLICMVersioning = MDNode::get(
+ Context, {MDString::get(Context, "llvm.loop.licm_versioning.disable")});
+ MDNode *DisableDistribution = MDNode::get(
+ Context,
+ {MDString::get(Context, "llvm.loop.distribute.enable"), FalseVal});
+ MDNode *NewLoopID =
+ MDNode::get(Context, {Dummy, DisableUnroll, DisableVectorize,
+ DisableLICMVersioning, DisableDistribution});
+ NewLoopID->replaceOperandWith(0, NewLoopID);
+ L.setLoopID(NewLoopID);
+}
+
+// Restructure \p L into its IRCE sub-ranges via LoopSplitUtils instead of
+// LoopConstrainer, folding eliminated checks in the main partition. The pre/
+// main/post boundaries mirror LoopConstrainer::run exactly (both directions).
+static bool constrainLoopWithLoopSplitUtils(
+ Loop *L, LoopInfo &LI, ScalarEvolution &SE, DominatorTree &DT,
+ const LoopStructure &LS, const LoopConstrainer::SubRanges &SR,
+ Type *RangeTy, function_ref<void(Loop *, bool)> LPMAddNewLoop,
+ SmallVectorImpl<InductiveRangeCheck> &RangeChecksToEliminate) {
+ // Defer a nested loop with an EH-pad side exit: cloning it per partition
+ // re-enters the enclosing loop through the pad, giving irreducible control
+ // flow. (Top-level EH exits leave to non-loop code and stay reducible.)
+ if (L->getParentLoop()) {
+ SmallVector<BasicBlock *, 4> ExitBlocks;
+ L->getExitBlocks(ExitBlocks);
+ for (BasicBlock *Exit : ExitBlocks)
+ if (Exit->isEHPad())
+ return false;
+ }
+
+ // Allow the uncomputable-trip-count fallback (stride>1 symbolic-bound loops
+ // with no computable trip count) and a truncated latch compare (gated by
+ // -irce-allow-narrow-latch, as in calculateSubRanges) so LoopSplitUtils
+ // covers the shapes IRCE constrains.
+ LoopSplitUtils LSU(L, &LI, &SE, &DT, /*AllowUncomputableTripCount=*/true,
+ /*AllowTruncatedLatchCompare=*/AllowNarrowLatchCondition);
+ if (!LSU.isLegal())
+ return false;
+ PHINode *IndPHI = LSU.getInductionVariable();
+ auto *IndTy = dyn_cast<IntegerType>(IndPHI->getType());
+ // The induction is the wide value; require the IRCE range type to match it
+ // (a range check narrower than the latch is rejected by calculateSubRanges).
+ if (!IndTy || IndTy != RangeTy)
+ return false;
+
+ const SCEV *StartS = SE.getSCEV(LS.IndVarStart);
+ if (StartS->getType() != IndTy) {
+ // Narrow-latch: widen LoopStructure's narrower-typed start to the induction/
+ // range type (mirroring LoopConstrainer's NoopOrExtend).
+ auto *StartTy = dyn_cast<IntegerType>(StartS->getType());
+ if (!StartTy || StartTy->getBitWidth() > IndTy->getBitWidth())
+ return false;
+ StartS = NoopOrExtend(StartS, IndTy, SE, LS.IsSignedPredicate);
+ }
+ if (StartS->getType() != IndTy)
+ return false;
+ // Inclusive end of the tiled iteration space: the last counted value in exact
+ // mode; with no computable trip count the invariant bound stands in (the final
+ // partition keeps the original latch, so its end is only a placeholder).
+ const SCEV *EndIncl = LSU.isUncomputableTripCountMode()
+ ? LSU.getInductionBound()
+ : LSU.getInductionEnd();
+ if (!EndIncl || EndIncl->getType() != IndTy)
+ return false;
+
+ bool HasLow = SR.LowLimit.has_value();
+ bool HasHigh = SR.HighLimit.has_value();
+ if (!HasLow && !HasHigh) {
+ // Safe range covers the whole loop: every check is redundant and there is
+ // no head/tail to peel, so no split is needed -- just fold in place. With
+ // nothing to fold, the loop is unchanged.
+ if (RangeChecksToEliminate.empty())
+ return false;
+ LLVMContext &Context = L->getHeader()->getContext();
+ for (InductiveRangeCheck &IRC : RangeChecksToEliminate) {
+ Use *U = IRC.getCheckUse();
+ Value *Folded = IRC.getPassingDirection() ? ConstantInt::getTrue(Context)
+ : ConstantInt::getFalse(Context);
+ U->set(Folded);
+ }
+ return true;
+ }
+
+ const SCEV *One = SE.getOne(IndTy);
+ const SCEV *Low = HasLow ? *SR.LowLimit : nullptr;
+ const SCEV *High = HasHigh ? *SR.HighLimit : nullptr;
+ if ((Low && Low->getType() != IndTy) || (High && High->getType() != IndTy))
+ return false;
+
+ unsigned MainIdx;
+ if (LS.IndVarIncreasing) {
+ const SCEV *MainStart = HasLow ? Low : StartS;
+ const SCEV *MainEnd = HasHigh ? SE.getMinusSCEV(High, One) : EndIncl;
+ if (HasLow)
+ LSU.addPartition(StartS, SE.getMinusSCEV(Low, One));
+ LSU.addPartition(MainStart, MainEnd);
+ if (HasHigh)
+ LSU.addPartition(High, EndIncl);
+ MainIdx = HasLow ? 1 : 0;
+ } else {
+ // Decreasing: LoopSplitUtils carries the runtime induction value across
+ // partitions, so boundaries need not land on the grid. Require a constant
+ // step so the no-overflow reasoning below and the clamp are well defined.
+ const auto *IndAR = dyn_cast<SCEVAddRecExpr>(SE.getSCEV(IndPHI));
+ if (!IndAR)
+ return false;
+ const auto *StepC = dyn_cast<SCEVConstant>(IndAR->getStepRecurrence(SE));
+ if (!StepC)
+ return false;
+
+ // The interior boundaries HighLimit-1 / LowLimit-1 must not underflow;
+ // require the same no-overflow proof LoopConstrainer uses (see its run()).
+ if (HasHigh && !cannotBeMinInLoop(High, L, SE, LS.IsSignedPredicate))
+ return false;
+ if (HasLow && !cannotBeMinInLoop(Low, L, SE, LS.IsSignedPredicate))
+ return false;
+ const SCEV *MainStart = HasHigh ? SE.getMinusSCEV(High, One) : StartS;
+ const SCEV *MainEnd = HasLow ? Low : EndIncl;
+ if (HasHigh)
+ LSU.addPartition(StartS, High);
+ LSU.addPartition(MainStart, MainEnd);
+ if (HasLow)
+ LSU.addPartition(SE.getMinusSCEV(Low, One), EndIncl);
+ MainIdx = HasHigh ? 1 : 0;
+ }
+
+ if (LSU.getNumPartitions() < 2)
+ return false;
+ if (!LSU.split())
+ return false;
+
+ // Fold the eliminated checks in the main partition only; the pre/post
+ // partitions keep the real checks because they run the unsafe head/tail.
+ LLVMContext &Context = L->getHeader()->getContext();
+ for (InductiveRangeCheck &IRC : RangeChecksToEliminate) {
+ Use *U = IRC.getCheckUse();
+ Value *Folded = IRC.getPassingDirection() ? ConstantInt::getTrue(Context)
+ : ConstantInt::getFalse(Context);
+ auto *User = cast<Instruction>(U->getUser());
+ unsigned OpNo = U->getOperandNo();
+ if (MainIdx == 0) {
+ User->setOperand(OpNo, Folded);
+ } else if (auto *ClonedUser = dyn_cast_or_null<Instruction>(
+ LSU.getPartitionValue(User, MainIdx))) {
+ ClonedUser->setOperand(OpNo, Folded);
+ }
+ }
+
+ // Match LoopConstrainer's nsw on the main-partition induction increment.
+ if (LS.IsSignedPredicate) {
+ Value *MainIndBase = MainIdx == 0
+ ? LS.IndVarBase
+ : LSU.getPartitionValue(LS.IndVarBase, MainIdx);
+ if (MainIndBase && isa<OverflowingBinaryOperator>(MainIndBase))
+ cast<BinaryOperator>(MainIndBase)->setHasNoSignedWrap(true);
+ }
+
+ // Give every non-main partition LoopConstrainer's pre/post treatment: disable
+ // further loop opts, tag the latch against reprocessing, register as sibling.
+ for (unsigned I = 0, E = LSU.getNumPartitions(); I != E; ++I) {
+ if (I == MainIdx)
+ continue;
+ Loop *PL = LSU.getPartitionLoop(I);
+ if (!PL)
+ continue;
+ disableAllLoopOptsOnLoop(*PL);
+ if (BasicBlock *Latch = PL->getLoopLatch())
+ Latch->getTerminator()->setMetadata(LoopConstrainerClonedLoopTag,
+ MDNode::get(Context, {}));
+ LPMAddNewLoop(PL, /*IsSubloop=*/false);
+ }
+
+ SE.forgetLoop(L);
+ return true;
+}
+
bool InductiveRangeCheckElimination::run(
Loop *L, function_ref<void(Loop *, bool)> LPMAddNewLoop) {
if (L->getBlocks().size() >= LoopSizeCutoff) {
@@ -1079,8 +1283,23 @@ bool InductiveRangeCheckElimination::run(
return false;
}
- LoopConstrainer LC(*L, LI, LPMAddNewLoop, LS, SE, DT,
- SafeIterRange->getBegin()->getType(), *MaybeSR);
+ Type *RangeTy = SafeIterRange->getBegin()->getType();
+
+ // -irce-use-loop-split-utils selects the engine: ON drives pre/main/post via
+ // LoopSplitUtils (folding checks itself; on decline the loop is left
+ // unconstrained, no fallback), OFF uses the legacy LoopConstrainer below.
+ if (UseLoopSplitUtils) {
+ if (constrainLoopWithLoopSplitUtils(L, LI, SE, DT, LS, *MaybeSR, RangeTy,
+ LPMAddNewLoop, RangeChecksToEliminate)) {
+ LLVM_DEBUG(dbgs() << "irce: constrained loop via LoopSplitUtils\n");
+ return true;
+ }
+ LLVM_DEBUG(dbgs() << "irce: LoopSplitUtils declined; leaving loop "
+ "unconstrained (no LoopConstrainer fallback)\n");
+ return Changed;
+ }
+
+ LoopConstrainer LC(*L, LI, LPMAddNewLoop, LS, SE, DT, RangeTy, *MaybeSR);
if (LC.run()) {
Changed = true;
diff --git a/llvm/test/Transforms/IRCE/loop-split-utils-driver-narrow-latch.ll b/llvm/test/Transforms/IRCE/loop-split-utils-driver-narrow-latch.ll
new file mode 100644
index 0000000000000..7e63303080df4
--- /dev/null
+++ b/llvm/test/Transforms/IRCE/loop-split-utils-driver-narrow-latch.ll
@@ -0,0 +1,39 @@
+; RUN: opt -passes=irce -irce-use-loop-split-utils=true -irce-allow-narrow-latch=true \
+; RUN: -irce-skip-profitability-checks -S < %s 2>&1 | FileCheck %s
+;
+; Narrow-latch shape: a wide i64 induction whose counted exit compares a
+; truncation of it ("icmp slt i32 (trunc iv.next), 100"). With
+; -irce-allow-narrow-latch, IRCE drives it through LoopSplitUtils'
+; AllowTruncatedLatchCompare path. The wide i64 induction drives the partition
+; boundaries and clamps (emitted in the wide type), the main loop folds the
+; range check to true, and the guarded post-loop keeps the original check.
+
+; CHECK-LABEL: @irce_driver_narrow_latch(
+; The main loop folds the range check and clamps the wide induction inclusively:
+; CHECK: loop:
+; CHECK: br i1 true, label %backedge, label %check_failed
+; CHECK: backedge:
+; CHECK: icmp sle i64 %iv.next, 98
+; A guarded post-loop keeps the original wide range check:
+; CHECK: ls.guard1:
+; CHECK: loop.ls1:
+; CHECK: %[[RC:.*]] = icmp slt i64 %iv.ls1, 99
+; CHECK: br i1 %[[RC]], label %backedge.ls1, label %check_failed
+
+define i32 @irce_driver_narrow_latch() {
+entry:
+ br label %loop
+loop:
+ %iv = phi i64 [ 0, %entry ], [ %iv.next, %backedge ]
+ %rc = icmp slt i64 %iv, 99
+ br i1 %rc, label %backedge, label %check_failed
+backedge:
+ %iv.next = add i64 %iv, 1
+ %narrow.iv = trunc i64 %iv.next to i32
+ %latch.cond = icmp slt i32 %narrow.iv, 100
+ br i1 %latch.cond, label %loop, label %exit
+exit:
+ ret i32 %narrow.iv
+check_failed:
+ ret i32 -1
+}
diff --git a/llvm/test/Transforms/IRCE/loop-split-utils-driver.ll b/llvm/test/Transforms/IRCE/loop-split-utils-driver.ll
new file mode 100644
index 0000000000000..8539c55dbfeb7
--- /dev/null
+++ b/llvm/test/Transforms/IRCE/loop-split-utils-driver.ll
@@ -0,0 +1,112 @@
+; RUN: opt -passes=irce -irce-use-loop-split-utils=true \
+; RUN: -irce-skip-profitability-checks -S < %s 2>&1 | FileCheck %s
+;
+; With -irce-use-loop-split-utils, IRCE restructures the loop through the
+; generic LoopSplitUtils primitive instead of LoopConstrainer. The result is
+; behaviorally identical -- the main range runs with the bounds check folded to
+; true, and a post range keeps the original check -- but the IR shape (block
+; names, block count) differs. This test pins those semantic invariants.
+
+; CHECK-LABEL: @irce_driver_upper(
+; The main loop runs with the range check folded to a constant true:
+; CHECK: loop:
+; CHECK: br i1 true, label %in.bounds, label %out.of.bounds
+; A guarded post range keeps the original (unfolded) range check:
+; CHECK: ls.guard1:
+; CHECK: loop.ls1:
+; CHECK: %[[UC:.*]] = icmp slt i32 %i.ls1, 30
+; CHECK: br i1 %[[UC]], label %in.bounds.ls1, label %out.of.bounds
+
+define i32 @irce_driver_upper(i32 %n) {
+entry:
+ %g = icmp sgt i32 %n, 0
+ br i1 %g, label %loop, label %early
+early:
+ ret i32 0
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %in.bounds ]
+ %acc = phi i32 [ 0, %entry ], [ %acc.next, %in.bounds ]
+ %i.next = add nsw i32 %i, 1
+ %uc = icmp slt i32 %i, 30
+ br i1 %uc, label %in.bounds, label %out.of.bounds
+in.bounds:
+ %m = mul i32 %i, 3
+ %acc.next = add i32 %acc, %m
+ %next = icmp slt i32 %i.next, %n
+ br i1 %next, label %loop, label %exit
+out.of.bounds:
+ br label %exit
+exit:
+ %r = phi i32 [ %acc.next, %in.bounds ], [ %acc, %out.of.bounds ]
+ ret i32 %r
+}
+
+; A decreasing loop (step -1) with a lower+upper range check: only the upper
+; limit is unsafe, so a pre-loop keeps the real check and the main partition
+; (the clone) folds it. The decreasing clamp uses an sge test.
+; CHECK-LABEL: @irce_driver_decreasing(
+; CHECK: ls.guard0:
+; CHECK: icmp sge i32 %{{.*}}, %smax
+; The main partition folds the range check to constant true:
+; CHECK: loop.ls1:
+; CHECK: and i1 true, true
+; CHECK: ls.final.exit:
+
+define void @irce_driver_decreasing(ptr %arr, ptr %a_len_ptr, i32 %n) {
+entry:
+ %len = load i32, ptr %a_len_ptr, !range !0
+ %first.itr.check = icmp sgt i32 %n, 0
+ %start = sub i32 %n, 1
+ br i1 %first.itr.check, label %loop, label %exit
+loop:
+ %idx = phi i32 [ %start, %entry ], [ %idx.dec, %in.bounds ]
+ %idx.dec = sub i32 %idx, 1
+ %abc.high = icmp slt i32 %idx, %len
+ %abc.low = icmp sge i32 %idx, 0
+ %abc = and i1 %abc.low, %abc.high
+ br i1 %abc, label %in.bounds, label %out.of.bounds
+in.bounds:
+ %addr = getelementptr i32, ptr %arr, i32 %idx
+ store i32 0, ptr %addr
+ %next = icmp sgt i32 %idx.dec, -1
+ br i1 %next, label %loop, label %exit
+out.of.bounds:
+ ret void
+exit:
+ ret void
+}
+
+; A non-unit stride (IV += 7) has no SCEV-computable trip count, so IRCE drives
+; it through LoopSplitUtils' uncomputable-trip-count fallback: the main loop
+; folds the check to true and clamps inclusively against a grid-aligned bound,
+; while the post-loop keeps the original check and the original latch.
+; CHECK-LABEL: @irce_driver_stride(
+; CHECK: loop:
+; CHECK: %idx.next = add nsw i32 %idx, 7
+; CHECK: br i1 true, label %in.bounds, label %out.of.bounds
+; CHECK: icmp sle i32 %idx.next, %{{.*}}
+; The post-loop keeps the original (unfolded) range check:
+; CHECK: loop.ls1:
+; CHECK: icmp slt i32 %idx.ls1, %len
+
+define void @irce_driver_stride(ptr %arr, ptr %a_len_ptr) {
+entry:
+ %len = load i32, ptr %a_len_ptr, !range !0
+ br label %loop
+loop:
+ %idx = phi i32 [ 0, %entry ], [ %idx.next, %in.bounds ]
+ %idx.next = add i32 %idx, 7
+ %abc = icmp slt i32 %idx, %len
+ br i1 %abc, label %in.bounds, label %out.of.bounds
+in.bounds:
+ %addr = getelementptr i32, ptr %arr, i32 %idx
+ store i32 0, ptr %addr
+ %next = icmp slt i32 %idx.next, 100
+ br i1 %next, label %loop, label %exit
+out.of.bounds:
+ ret void
+exit:
+ ret void
+}
+
+!0 = !{i32 0, i32 2147483647}
>From c20bb37f144ac9204822860cb2be33db117830e0 Mon Sep 17 00:00:00 2001
From: Ashutosh Nema <ashu1212 at gmail.com>
Date: Mon, 13 Jul 2026 16:31:41 +0530
Subject: [PATCH 6/6] [LoopBoundSplit] Optionally transform via LoopSplitUtils
(default off, NFC)
Add -loop-bound-split-use-loop-split-utils (hidden, default off): when set,
LoopBoundSplit performs its split through the generic LoopSplitUtils. The split
bound is rounded up to the next induction-grid value, the guardless pre-loop and
the post-loop partitions are created via the utility, and the split condition is
folded true/false in the pre/post loops. LCSSA/loop-simplify are rebuilt before
handing the post-loop to the pass manager. With the option off there is no
behavioural change. Adds loop-split-utils-driver.ll.
---
llvm/lib/Transforms/Scalar/LoopBoundSplit.cpp | 95 +++++++++++++++++++
.../LoopBoundSplit/loop-split-utils-driver.ll | 51 ++++++++++
2 files changed, 146 insertions(+)
create mode 100644 llvm/test/Transforms/LoopBoundSplit/loop-split-utils-driver.ll
diff --git a/llvm/lib/Transforms/Scalar/LoopBoundSplit.cpp b/llvm/lib/Transforms/Scalar/LoopBoundSplit.cpp
index 27ccce3328d27..9c97cc3404d0b 100644
--- a/llvm/lib/Transforms/Scalar/LoopBoundSplit.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopBoundSplit.cpp
@@ -13,10 +13,13 @@
#include "llvm/Analysis/ScalarEvolution.h"
#include "llvm/Analysis/ScalarEvolutionExpressions.h"
#include "llvm/IR/PatternMatch.h"
+#include "llvm/Support/CommandLine.h"
#include "llvm/Transforms/Scalar/LoopPassManager.h"
#include "llvm/Transforms/Utils/BasicBlockUtils.h"
#include "llvm/Transforms/Utils/Cloning.h"
#include "llvm/Transforms/Utils/LoopSimplify.h"
+#include "llvm/Transforms/Utils/LoopSplitUtils.h"
+#include "llvm/Transforms/Utils/LoopUtils.h"
#include "llvm/Transforms/Utils/ScalarEvolutionExpander.h"
#define DEBUG_TYPE "loop-bound-split"
@@ -24,6 +27,12 @@
using namespace llvm;
using namespace PatternMatch;
+// When enabled, the bound split runs through the generic LoopSplitUtils instead
+// of this pass's bespoke clone-and-fold transform (equivalent, different IR).
+static cl::opt<bool> UseLoopSplitUtils(
+ "loop-bound-split-use-loop-split-utils", cl::init(false), cl::Hidden,
+ cl::desc("Drive LoopBoundSplit's transform through LoopSplitUtils"));
+
namespace {
struct ConditionInfo {
/// Branch instruction with this condition
@@ -283,6 +292,87 @@ static CondBrInst *findSplitCandidate(const Loop &L, ScalarEvolution &SE,
return nullptr;
}
+// Opt-in transform: cut the iteration space where the split condition flips,
+// then fold it (true in the pre-loop, false in the post-loop). Equivalent to
+// the default transform below, but emits LoopSplitUtils' canonical IR shape.
+static bool
+splitLoopBoundWithLoopSplitUtils(Loop &L, DominatorTree &DT, LoopInfo &LI,
+ ScalarEvolution &SE, LPMUpdater &U,
+ ConditionInfo &SplitCandidateCond) {
+ LoopSplitUtils LSU(&L, &LI, &SE, &DT);
+ if (!LSU.isLegal())
+ return false;
+
+ const auto *IndAR =
+ dyn_cast<SCEVAddRecExpr>(SE.getSCEV(LSU.getInductionVariable()));
+ if (!IndAR || !IndAR->isAffine())
+ return false;
+
+ // The split condition must be over the loop's induction recurrence so its
+ // boundary maps directly onto the iteration space the utility partitions.
+ if (SplitCandidateCond.AddRecSCEV != IndAR)
+ return false;
+
+ Type *IndTy = IndAR->getType();
+ const SCEV *SplitBoundSCEV = SplitCandidateCond.BoundSCEV;
+ if (SplitBoundSCEV->getType() != IndTy)
+ return false;
+
+ // The split condition is normalized to "AddRec < SplitBound", so the pre-loop
+ // covers iterations with induction < SplitBound and the post-loop the rest.
+ // findSplitCandidate guarantees SplitBound > start, so the pre-loop is nonempty.
+ const SCEV *Step = IndAR->getStepRecurrence(SE);
+ const SCEV *Start0 = IndAR->getStart();
+
+ // Round SplitBound up to the next induction-grid value {Start + k*Step} --
+ // Start1 = Start + Step*ceil((SplitBound-Start)/Step) -- the first iteration
+ // "iv < SplitBound" is false for. Step>0 and SplitBound>Start make it exact.
+ const auto *StepC = dyn_cast<SCEVConstant>(Step);
+ if (!StepC || !StepC->getAPInt().isStrictlyPositive())
+ return false;
+ const SCEV *Delta = SE.getMinusSCEV(SplitBoundSCEV, Start0);
+ const SCEV *K = SE.getUDivExpr(
+ SE.getAddExpr(Delta, SE.getMinusSCEV(Step, SE.getOne(IndTy))), Step);
+ const SCEV *Start1 = SE.getAddExpr(Start0, SE.getMulExpr(K, Step));
+ const SCEV *End0 = SE.getMinusSCEV(Start1, Step);
+ const SCEV *BTC = SE.getBackedgeTakenCount(&L);
+ const SCEV *End1 = IndAR->evaluateAtIteration(BTC, SE);
+ if (End1->getType() != IndTy)
+ return false;
+
+ LSU.addPartition(Start0, End0);
+ LSU.addPartition(Start1, End1);
+ LSU.avoidPartitionGuard(0);
+
+ if (!LSU.split())
+ return false;
+
+ // Fold the split condition: true in the pre-loop (partition 0), false in the
+ // post-loop (partition 1). The dead arm is cleaned up later (SimplifyCFG).
+ LLVMContext &Context = L.getHeader()->getContext();
+ SplitCandidateCond.BI->setCondition(ConstantInt::getTrue(Context));
+ if (auto *ClonedSplitCandidateBI = dyn_cast_or_null<CondBrInst>(
+ LSU.getPartitionValue(SplitCandidateCond.BI, 1)))
+ ClonedSplitCandidateBI->setCondition(ConstantInt::getFalse(Context));
+
+ // Refresh analyses and hand the post-loop to the pass manager. The splitter's
+ // SSA fix-up does not restore LCSSA across the new boundaries, so rebuild it
+ // before re-simplifying (simplifyLoop asserts LCSSA when preserving it).
+ SE.forgetLoop(&L);
+ Loop *PostLoop = LSU.getPartitionLoop(1);
+ formLCSSARecursively(L, DT, &LI, &SE);
+ if (PostLoop)
+ formLCSSARecursively(*PostLoop, DT, &LI, &SE);
+ simplifyLoop(&L, &DT, &LI, &SE, nullptr, nullptr, /*PreserveLCSSA=*/true);
+ if (PostLoop) {
+ simplifyLoop(PostLoop, &DT, &LI, &SE, nullptr, nullptr,
+ /*PreserveLCSSA=*/true);
+ U.addSiblingLoops(PostLoop);
+ }
+
+ return true;
+}
+
static bool splitLoopBound(Loop &L, DominatorTree &DT, LoopInfo &LI,
ScalarEvolution &SE, LPMUpdater &U) {
ConditionInfo SplitCandidateCond;
@@ -298,6 +388,11 @@ static bool splitLoopBound(Loop &L, DominatorTree &DT, LoopInfo &LI,
if (!isProfitableToTransform(L, SplitCandidateCond.BI))
return false;
+ // Opt-in: perform the transform through the generic splitter instead.
+ if (UseLoopSplitUtils)
+ return splitLoopBoundWithLoopSplitUtils(L, DT, LI, SE, U,
+ SplitCandidateCond);
+
// Now, we have a split candidate. Let's build a form as below.
// +--------------------+
// | preheader |
diff --git a/llvm/test/Transforms/LoopBoundSplit/loop-split-utils-driver.ll b/llvm/test/Transforms/LoopBoundSplit/loop-split-utils-driver.ll
new file mode 100644
index 0000000000000..67723d2a90f23
--- /dev/null
+++ b/llvm/test/Transforms/LoopBoundSplit/loop-split-utils-driver.ll
@@ -0,0 +1,51 @@
+; RUN: opt -passes=loop-bound-split -verify-each \
+; RUN: -loop-bound-split-use-loop-split-utils=true -S < %s | FileCheck %s
+;
+; With -loop-bound-split-use-loop-split-utils, LoopBoundSplit performs the split
+; through the generic LoopSplitUtils primitive instead of its bespoke
+; clone-and-fold transform. The result is behaviorally identical -- a pre-loop
+; that runs while the split condition holds (folded to true) and a guarded
+; post-loop that runs the rest (folded to false) -- but emits LoopSplitUtils'
+; canonical ls.guard/ls.exit block shape. This test pins those invariants.
+
+; CHECK-LABEL: @split_bound(
+; The pre-loop's split condition is folded to a constant true, and its latch is
+; clamped to the split boundary (min(n-1, 511)):
+; CHECK: loop:
+; CHECK: br i1 true, label %then, label %else
+; CHECK: latch:
+; CHECK: %[[IN:.*]] = add nsw i32 %i, 1
+; CHECK: icmp sle i32 %[[IN]], %smin
+; A guard decides whether the post-loop runs at all:
+; CHECK: ls.guard1:
+; CHECK: icmp sle i32 512, %{{.*}}
+; The post-loop's split condition is folded to a constant false:
+; CHECK: loop.ls1:
+; CHECK: br i1 false, label %then.ls1, label %else.ls1
+; CHECK: ls.final.exit:
+; CHECK: ret void
+
+; for (i = 0; i < n; i++) { if (i < 512) a[i] = i*3; else a[i] = 7; }
+define void @split_bound(ptr %a, i32 %n) {
+entry:
+ br label %loop
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %latch ]
+ %c = icmp slt i32 %i, 512
+ br i1 %c, label %then, label %else
+then:
+ %m = mul i32 %i, 3
+ %p0 = getelementptr inbounds i32, ptr %a, i32 %i
+ store i32 %m, ptr %p0
+ br label %latch
+else:
+ %p1 = getelementptr inbounds i32, ptr %a, i32 %i
+ store i32 7, ptr %p1
+ br label %latch
+latch:
+ %i.next = add nsw i32 %i, 1
+ %cond = icmp slt i32 %i.next, %n
+ br i1 %cond, label %loop, label %exit
+exit:
+ ret void
+}
More information about the llvm-commits
mailing list