[llvm] [LoopInterchange] Consider eligible inner subnests (PR #214920)
via llvm-commits
llvm-commits at lists.llvm.org
Fri Aug 7 19:56:35 PDT 2026
https://github.com/MattPD created https://github.com/llvm/llvm-project/pull/214920
LoopInterchange considers the complete outermost loop nest. When that nest is
unsuitable for the existing whole-nest path, the pass gives up even when a
parent loop and its leaf child form a legal, profitable pair. Add a default-on
fallback that considers those pairs.
Before adding the fallback, harden two shared checks. Compare
`MaxMemInstrRatio * NumInsts` with `NumMemInstr * NumMemInstr` in 64 bits.
Wrapped products can reject a nest spuriously or bypass the bail-out that
bounds dependence analysis. Reject a dependence direction vector with more
levels than the analyzed loop chain before padding shorter vectors.
Use `LoopNest::getNestDepth()` for the maximum-depth cutoff. The old cutoff
used the size of `LoopNest::getLoops()`, which lists every descendant in
breadth-first order. Sibling loops could therefore make a shallow nest appear
too deep. This correction applies independently of the fallback.
The fallback walks the tree once and carries each loop's depth. It considers
direct parent and leaf-child pairs deepest first, then transforms at most one
legal, profitable pair. It retains at most ten attempts by default through
`-loop-interchange-max-inner-subnest-candidates`, preserves the complete
ancestor dependence prefix, excludes sibling-loop memory accesses, and
preserves `LoopInfo` sibling order. Cache analysis uses the candidate's
root-to-leaf ancestor chain instead of the whole tree.
Non-linear nests route directly to the fallback and no longer emit the
whole-nest `Dependence` analysis remark whose text begins
`Computed dependence info`. The hidden
`-loop-interchange-enable-inner-subnest-fallback` option defaults to true.
Setting it to false disables only the fallback transformation; it does not
restore the previous depth cutoff or whole-nest remark.
Assisted-by: Claude Opus 4.8, Claude Opus 5, Claude Sonnet 5, GPT-5.6 Sol.
>From be9e0ff438f4864001428eb9f7d23e9e8a1b0dd9 Mon Sep 17 00:00:00 2001
From: "Matt P. Dziubinski" <matt-p.dziubinski at hpe.com>
Date: Fri, 7 Aug 2026 02:43:23 -0500
Subject: [PATCH 1/3] [LoopInterchange] Harden dependence-analysis guards
Compare the memory-ratio products in 64 bits. The ratio product could
wrap and spuriously reject a nest. The squared memory-instruction count
could also wrap and bypass the dependence-analysis guard.
Reject direction vectors longer than the analyzed chain before padding.
Without that guard, an unexpected longer vector makes the padding loop never
terminate.
Add a large-ratio behavior test for the spurious-rejection failure mode.
Assisted-by: Claude Opus 5, GPT-5.6 Sol.
---
.../lib/Transforms/Scalar/LoopInterchange.cpp | 21 ++++++++++++++++---
.../LoopInterchange/memory-instr-ratio.ll | 5 +++++
2 files changed, 23 insertions(+), 3 deletions(-)
diff --git a/llvm/lib/Transforms/Scalar/LoopInterchange.cpp b/llvm/lib/Transforms/Scalar/LoopInterchange.cpp
index b6e8b0efd7325..e667decf7149c 100644
--- a/llvm/lib/Transforms/Scalar/LoopInterchange.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopInterchange.cpp
@@ -173,7 +173,7 @@ static bool populateDependencyMatrix(CharMatrix &DepMatrix, unsigned Level,
using ValueVector = SmallVector<Value *, 16>;
ValueVector MemInstr;
- unsigned NumInsts = 0;
+ uint64_t NumInsts = 0;
// For each block.
for (BasicBlock *BB : L->blocks()) {
@@ -199,10 +199,13 @@ static bool populateDependencyMatrix(CharMatrix &DepMatrix, unsigned Level,
// instructions. On the other hand, if the number of memory instructions is
// not small, but the loop is large (i.e., it contains many non-memory
// instructions), the analysis can still be affordable.
- unsigned NumMemInstr = MemInstr.size();
+ uint64_t NumMemInstr = MemInstr.size();
LLVM_DEBUG(dbgs() << "Found " << NumMemInstr
<< " Loads and Stores to analyze\n");
- if (MaxMemInstrRatio * NumInsts < NumMemInstr * NumMemInstr) {
+ // Compare in 64 bits: in 32-bit arithmetic the ratio product could wrap and
+ // spuriously reject, while the squared memory count could bypass this guard.
+ if (static_cast<uint64_t>(MaxMemInstrRatio) * NumInsts <
+ NumMemInstr * NumMemInstr) {
ORE->emit([&]() {
return OptimizationRemarkMissed(DEBUG_TYPE, "UnsupportedLoop",
L->getStartLoc(), L->getHeader())
@@ -265,6 +268,18 @@ static bool populateDependencyMatrix(CharMatrix &DepMatrix, unsigned Level,
Dep.assign(Level, '*');
}
+ // A direction vector longer than the analyzed nesting depth cannot be
+ // represented in this matrix. The invariant is asserted because both
+ // callers analyze complete chains. Retain a conservative release-mode
+ // bail for any future caller that violates it.
+ assert(Dep.size() <= Level &&
+ "Direction vector is deeper than the analyzed nest");
+ if (Dep.size() > Level) {
+ LLVM_DEBUG(dbgs() << "Direction vector is longer than the analyzed "
+ "nesting depth; rejecting.\n");
+ return false;
+ }
+
while (Dep.size() != Level) {
Dep.push_back('I');
}
diff --git a/llvm/test/Transforms/LoopInterchange/memory-instr-ratio.ll b/llvm/test/Transforms/LoopInterchange/memory-instr-ratio.ll
index 7f76f25612d03..e623a41020134 100644
--- a/llvm/test/Transforms/LoopInterchange/memory-instr-ratio.ll
+++ b/llvm/test/Transforms/LoopInterchange/memory-instr-ratio.ll
@@ -3,6 +3,11 @@
; RUN: -loop-interchange-max-mem-instr-ratio=1 | FileCheck %s --check-prefixes=CHECK,CHECK-RATIO-1
; RUN: opt < %s -passes=loop-interchange -S -loop-interchange-profitabilities=ignore \
; RUN: -loop-interchange-max-mem-instr-ratio=100 | FileCheck %s --check-prefixes=CHECK,CHECK-RATIO-100
+; The outer loop's blocks contain 20 instructions. Keep that count even:
+; 2^31 times an even count wraps to zero in 32 bits and spuriously rejects this
+; otherwise eligible nest. An odd count would silently stop testing the wrap.
+; RUN: opt < %s -passes=loop-interchange -S -loop-interchange-profitabilities=ignore \
+; RUN: -loop-interchange-max-mem-instr-ratio=2147483648 | FileCheck %s --check-prefixes=CHECK,CHECK-RATIO-100
define void @f(ptr noalias %A) {
; CHECK-RATIO-1-LABEL: define void @f(
>From 64a497a6ac08e57082e5a2956cf75ff1aedceaed Mon Sep 17 00:00:00 2001
From: "Matt P. Dziubinski" <matt-p.dziubinski at hpe.com>
Date: Fri, 7 Aug 2026 02:43:51 -0500
Subject: [PATCH 2/3] [LoopInterchange] Add inner-subnest candidate tests
Add tests for profitable inner pairs currently suppressed by sibling-rich,
uncomputable, or non-linear enclosing nests. Preserve strict FP,
structure, and dependence-context negatives for the follow-up candidate
formation change.
Assisted-by: Claude Opus 4.8, Claude Sonnet 5, GPT-5.6 Sol.
---
.../inner-subnest-candidates.ll | 962 ++++++++++++++++++
.../inner-subnest-dependences.ll | 414 ++++++++
2 files changed, 1376 insertions(+)
create mode 100644 llvm/test/Transforms/LoopInterchange/inner-subnest-candidates.ll
create mode 100644 llvm/test/Transforms/LoopInterchange/inner-subnest-dependences.ll
diff --git a/llvm/test/Transforms/LoopInterchange/inner-subnest-candidates.ll b/llvm/test/Transforms/LoopInterchange/inner-subnest-candidates.ll
new file mode 100644
index 0000000000000..0fc41f3f0645f
--- /dev/null
+++ b/llvm/test/Transforms/LoopInterchange/inner-subnest-candidates.ll
@@ -0,0 +1,962 @@
+; Precommit test for inner-subnest candidate formation in LoopInterchange.
+;
+; This file is entirely hand-maintained. Do NOT run update_test_checks.py on
+; it: the CHECK lines deliberately pin the *current* no-transform behavior of
+; the unmodified LoopInterchange pass so that the follow-on
+; candidate-formation implementation can show a reviewable diff.
+;
+; Background:
+; LoopInterchangePass::run consumes LoopNest::getLoops(), which is a
+; *breadth-first* walk over every descendant loop (siblings included). The
+; pass then (1) rejects when that flat list is longer than
+; MaxLoopNestDepth (=10), (2) rejects when any member has an uncomputable
+; backedge / non-unique exit, and (3) in LoopInterchange::run(LoopNest&)
+; rejects when the flat list is not a single linear chain. In each of those
+; cases the pass bails *before* it ever reaches an otherwise eligible,
+; profitable adjacent parent/child pair, so no interchange happens today.
+;
+; A fixed leading dimension of 1335 doubles gives a 10,680-byte inner stride
+; (1335 * 8), a cache-hostile column-major access typical of shallow-water
+; stencil benchmarks. The two `admitted_*` functions below are standalone,
+; admissible 2-deep nests that the *current* pass already interchanges under
+; default profitability; they establish that the shared candidate pair is legal
+; and profitable "once admitted". Every other function embeds that same shape
+; (or a deliberately-permanent negative) inside an enclosing structure that the
+; current pass rejects, and pins that the pair is left in its original order.
+;
+; This precommit only asserts behavior observable on the unmodified pass: the
+; breadth-first depth remark, the applied/analysis/missed remarks, and the
+; actual unswapped IR. Direct-edge candidate selection, ancestor-column
+; dependence handling and sibling exclusion are oracles for the follow-on
+; candidate-formation implementation and are intentionally NOT asserted here.
+;
+; RUN: opt < %s -passes=loop-interchange -cache-line-size=64 \
+; RUN: -verify-dom-info -verify-loop-info -verify-scev -verify-loop-lcssa \
+; RUN: -S 2>&1 | FileCheck %s --check-prefix=IR
+;
+; Full, function-associated remark log (Passed / Missed / Analysis).
+; RUN: opt < %s -passes=loop-interchange -cache-line-size=64 \
+; RUN: -pass-remarks=loop-interchange -pass-remarks-missed=loop-interchange \
+; RUN: -pass-remarks-output=%t -disable-output
+; RUN: FileCheck %s --check-prefix=YAML --input-file=%t
+;
+; Raising the depth cap past the breadth-first count removes the (false) depth
+; rejection for the shallow, sibling-rich nest, proving the count -- not a real
+; depth-3 problem -- triggered it. Under the raised cap bfs_loop_count_is_not_depth
+; clears the depth gate and reaches dependence analysis (its own function-
+; associated !Analysis Dependence record) instead of UnsupportedLoopNestDepth; it
+; still does not interchange (its next blocker is non-linearity), which the
+; follow-on candidate-formation implementation addresses. Proven from the
+; function-associated YAML remark stream, not a shared stderr string that
+; another function could satisfy.
+; RUN: opt < %s -passes=loop-interchange -cache-line-size=64 \
+; RUN: -loop-interchange-max-loop-nest-depth=32 \
+; RUN: -pass-remarks-output=%t.raised -disable-output
+; RUN: FileCheck %s --check-prefix=RAISED --input-file=%t.raised \
+; RUN: --implicit-check-not=UnsupportedLoopNestDepth
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+
+;-------------------------------------------------------------------------------
+; Expected current remark log (function/module order). See per-function notes.
+;-------------------------------------------------------------------------------
+; YAML: --- !Analysis
+; YAML: Name: Dependence
+; YAML: Function: admitted_1335_pair_one_reduction
+; YAML: --- !Passed
+; YAML: Name: Interchanged
+; YAML: Function: admitted_1335_pair_one_reduction
+; YAML: --- !Analysis
+; YAML: Name: Dependence
+; YAML: Function: admitted_1335_pair_three_reductions
+; YAML: --- !Passed
+; YAML: Name: Interchanged
+; YAML: Function: admitted_1335_pair_three_reductions
+; YAML: --- !Missed
+; YAML: Name: UnsupportedLoopNestDepth
+; YAML: Function: bfs_loop_count_is_not_depth
+; The two uncomputable-neighbour nests bail before the analysis remark, so they
+; emit nothing at all between the depth miss and the two-candidate analysis.
+; YAML-NOT: Function: uncomputable_sibling_does_not_block
+; YAML-NOT: Function: uncomputable_ancestor_partition
+; YAML: --- !Analysis
+; YAML: Name: Dependence
+; YAML: Function: two_candidate_pairs_one_fallback
+; YAML: --- !Analysis
+; YAML: Name: Dependence
+; YAML: Function: partly_exact_reduction
+; YAML: --- !Missed
+; YAML: Name: UnsupportedPHIOuter
+; YAML: Function: partly_exact_reduction
+; YAML: --- !Analysis
+; YAML: Name: Dependence
+; YAML: Function: dynamic_leading_dimension_subnest
+; YAML: --- !Analysis
+; YAML: Name: Dependence
+; YAML: Function: all_exact_reduction_subnest
+; YAML: --- !Analysis
+; YAML: Name: Dependence
+; YAML: Function: partly_exact_reduction_subnest
+; YAML: --- !Analysis
+; YAML: Name: Dependence
+; YAML: Function: non_leaf_candidate_subnest
+
+; The two admitted controls still interchange (two Interchanged records); the very
+; next analysis record is bfs_loop_count_is_not_depth, proving it now clears the
+; raised depth gate and reaches dependence analysis rather than being rejected as
+; too deep. --implicit-check-not proves no UnsupportedLoopNestDepth is emitted.
+; RAISED: Name: Interchanged
+; RAISED: Name: Interchanged
+; RAISED: Name: Dependence
+; RAISED-NEXT: Function: bfs_loop_count_is_not_depth
+
+;-------------------------------------------------------------------------------
+; Positive controls: the shared fixed-1335 reduction pair, presented as a plain
+; admissible 2-deep nest, is interchanged by the current pass under default
+; profitability. These prove the pair is legal + profitable "once admitted", so
+; every blocked case below fails only because of its enclosing structure.
+; Their transformed IR is not pinned here (that belongs to the follow-on
+; behavior commit's before/after); the Passed remark above is the oracle.
+;-------------------------------------------------------------------------------
+
+; double sum = 0; for i: for j: sum += A[j][i]; (inner j strides 1335 doubles)
+define void @admitted_1335_pair_one_reduction(ptr %A, ptr %R) {
+entry:
+ br label %outer.header
+
+outer.header:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %outer.latch ]
+ %sum.i = phi double [ 0.000000e+00, %entry ], [ %sum.i.lcssa, %outer.latch ]
+ br label %inner.header
+
+inner.header:
+ %j = phi i64 [ 0, %outer.header ], [ %j.next, %inner.header ]
+ %sum.j = phi double [ %sum.i, %outer.header ], [ %sum.j.next, %inner.header ]
+ %idx = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+ %a = load double, ptr %idx, align 8
+ %sum.j.next = fadd reassoc double %sum.j, %a
+ %j.next = add i64 %j, 1
+ %j.ec = icmp eq i64 %j.next, 1335
+ br i1 %j.ec, label %outer.latch, label %inner.header
+
+outer.latch:
+ %sum.i.lcssa = phi double [ %sum.j.next, %inner.header ]
+ %i.next = add i64 %i, 1
+ %i.ec = icmp eq i64 %i.next, 1335
+ br i1 %i.ec, label %exit, label %outer.header
+
+exit:
+ %sum.res = phi double [ %sum.i.lcssa, %outer.latch ]
+ store double %sum.res, ptr %R, align 8
+ ret void
+}
+
+; Three independent reassociated reductions over three arrays, the shape of a
+; multi-array checksum loop.
+define void @admitted_1335_pair_three_reductions(ptr %A, ptr %B, ptr %C, ptr %R) {
+entry:
+ br label %outer.header
+
+outer.header:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %outer.latch ]
+ %sumA.i = phi double [ 0.000000e+00, %entry ], [ %sumA.i.lcssa, %outer.latch ]
+ %sumB.i = phi double [ 0.000000e+00, %entry ], [ %sumB.i.lcssa, %outer.latch ]
+ %sumC.i = phi double [ 0.000000e+00, %entry ], [ %sumC.i.lcssa, %outer.latch ]
+ br label %inner.header
+
+inner.header:
+ %j = phi i64 [ 0, %outer.header ], [ %j.next, %inner.header ]
+ %sumA.j = phi double [ %sumA.i, %outer.header ], [ %sumA.j.next, %inner.header ]
+ %sumB.j = phi double [ %sumB.i, %outer.header ], [ %sumB.j.next, %inner.header ]
+ %sumC.j = phi double [ %sumC.i, %outer.header ], [ %sumC.j.next, %inner.header ]
+ %idxA = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+ %a = load double, ptr %idxA, align 8
+ %sumA.j.next = fadd reassoc double %sumA.j, %a
+ %idxB = getelementptr inbounds [1335 x double], ptr %B, i64 %j, i64 %i
+ %b = load double, ptr %idxB, align 8
+ %sumB.j.next = fadd reassoc double %sumB.j, %b
+ %idxC = getelementptr inbounds [1335 x double], ptr %C, i64 %j, i64 %i
+ %c = load double, ptr %idxC, align 8
+ %sumC.j.next = fadd reassoc double %sumC.j, %c
+ %j.next = add i64 %j, 1
+ %j.ec = icmp eq i64 %j.next, 1335
+ br i1 %j.ec, label %outer.latch, label %inner.header
+
+outer.latch:
+ %sumA.i.lcssa = phi double [ %sumA.j.next, %inner.header ]
+ %sumB.i.lcssa = phi double [ %sumB.j.next, %inner.header ]
+ %sumC.i.lcssa = phi double [ %sumC.j.next, %inner.header ]
+ %i.next = add i64 %i, 1
+ %i.ec = icmp eq i64 %i.next, 1335
+ br i1 %i.ec, label %exit, label %outer.header
+
+exit:
+ %sumA.res = phi double [ %sumA.i.lcssa, %outer.latch ]
+ %sumB.res = phi double [ %sumB.i.lcssa, %outer.latch ]
+ %sumC.res = phi double [ %sumC.i.lcssa, %outer.latch ]
+ %rB = getelementptr inbounds double, ptr %R, i64 1
+ %rC = getelementptr inbounds double, ptr %R, i64 2
+ store double %sumA.res, ptr %R, align 8
+ store double %sumB.res, ptr %rB, align 8
+ store double %sumC.res, ptr %rC, align 8
+ ret void
+}
+
+;-------------------------------------------------------------------------------
+; (1) Sibling-rich, genuinely shallow nest (true depth 3) whose breadth-first
+; descendant count is 12 (top + pairX.outer + pairX.inner + sib1..sib9). The
+; current pass reports the flat count as an unsupported "depth" and never
+; considers the eligible fixed-1335 three-reduction pairX.outer/pairX.inner.
+;-------------------------------------------------------------------------------
+define void @bfs_loop_count_is_not_depth(ptr %A, ptr %B, ptr %C, ptr %R) {
+entry:
+ br label %top.header
+
+top.header:
+ %t = phi i64 [ 0, %entry ], [ %t.next, %top.latch ]
+ br label %pairX.outer.header
+
+pairX.outer.header:
+ %i = phi i64 [ 0, %top.header ], [ %i.next, %pairX.outer.latch ]
+ %sumA.i = phi double [ 0.000000e+00, %top.header ], [ %sumA.i.lcssa, %pairX.outer.latch ]
+ %sumB.i = phi double [ 0.000000e+00, %top.header ], [ %sumB.i.lcssa, %pairX.outer.latch ]
+ %sumC.i = phi double [ 0.000000e+00, %top.header ], [ %sumC.i.lcssa, %pairX.outer.latch ]
+ br label %pairX.inner
+
+pairX.inner:
+ %j = phi i64 [ 0, %pairX.outer.header ], [ %j.next, %pairX.inner ]
+ %sumA.j = phi double [ %sumA.i, %pairX.outer.header ], [ %sumA.j.next, %pairX.inner ]
+ %sumB.j = phi double [ %sumB.i, %pairX.outer.header ], [ %sumB.j.next, %pairX.inner ]
+ %sumC.j = phi double [ %sumC.i, %pairX.outer.header ], [ %sumC.j.next, %pairX.inner ]
+ %idxA = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+ %a = load double, ptr %idxA, align 8
+ %sumA.j.next = fadd reassoc double %sumA.j, %a
+ %idxB = getelementptr inbounds [1335 x double], ptr %B, i64 %j, i64 %i
+ %b = load double, ptr %idxB, align 8
+ %sumB.j.next = fadd reassoc double %sumB.j, %b
+ %idxC = getelementptr inbounds [1335 x double], ptr %C, i64 %j, i64 %i
+ %c = load double, ptr %idxC, align 8
+ %sumC.j.next = fadd reassoc double %sumC.j, %c
+ %j.next = add i64 %j, 1
+ %j.ec = icmp eq i64 %j.next, 1335
+ br i1 %j.ec, label %pairX.outer.latch, label %pairX.inner
+
+pairX.outer.latch:
+ %sumA.i.lcssa = phi double [ %sumA.j.next, %pairX.inner ]
+ %sumB.i.lcssa = phi double [ %sumB.j.next, %pairX.inner ]
+ %sumC.i.lcssa = phi double [ %sumC.j.next, %pairX.inner ]
+ %i.next = add i64 %i, 1
+ %i.ec = icmp eq i64 %i.next, 1335
+ br i1 %i.ec, label %pairX.exit, label %pairX.outer.header
+
+pairX.exit:
+ %sumA.live = phi double [ %sumA.i.lcssa, %pairX.outer.latch ]
+ %sumB.live = phi double [ %sumB.i.lcssa, %pairX.outer.latch ]
+ %sumC.live = phi double [ %sumC.i.lcssa, %pairX.outer.latch ]
+ %rB = getelementptr inbounds double, ptr %R, i64 1
+ %rC = getelementptr inbounds double, ptr %R, i64 2
+ store double %sumA.live, ptr %R, align 8
+ store double %sumB.live, ptr %rB, align 8
+ store double %sumC.live, ptr %rC, align 8
+ br label %sib1.header
+
+sib1.header:
+ %s1 = phi i64 [ 0, %pairX.exit ], [ %s1.next, %sib1.header ]
+ %s1.next = add i64 %s1, 1
+ %s1.ec = icmp eq i64 %s1.next, 4
+ br i1 %s1.ec, label %sib1.exit, label %sib1.header
+
+sib1.exit:
+ br label %sib2.header
+
+sib2.header:
+ %s2 = phi i64 [ 0, %sib1.exit ], [ %s2.next, %sib2.header ]
+ %s2.next = add i64 %s2, 1
+ %s2.ec = icmp eq i64 %s2.next, 4
+ br i1 %s2.ec, label %sib2.exit, label %sib2.header
+
+sib2.exit:
+ br label %sib3.header
+
+sib3.header:
+ %s3 = phi i64 [ 0, %sib2.exit ], [ %s3.next, %sib3.header ]
+ %s3.next = add i64 %s3, 1
+ %s3.ec = icmp eq i64 %s3.next, 4
+ br i1 %s3.ec, label %sib3.exit, label %sib3.header
+
+sib3.exit:
+ br label %sib4.header
+
+sib4.header:
+ %s4 = phi i64 [ 0, %sib3.exit ], [ %s4.next, %sib4.header ]
+ %s4.next = add i64 %s4, 1
+ %s4.ec = icmp eq i64 %s4.next, 4
+ br i1 %s4.ec, label %sib4.exit, label %sib4.header
+
+sib4.exit:
+ br label %sib5.header
+
+sib5.header:
+ %s5 = phi i64 [ 0, %sib4.exit ], [ %s5.next, %sib5.header ]
+ %s5.next = add i64 %s5, 1
+ %s5.ec = icmp eq i64 %s5.next, 4
+ br i1 %s5.ec, label %sib5.exit, label %sib5.header
+
+sib5.exit:
+ br label %sib6.header
+
+sib6.header:
+ %s6 = phi i64 [ 0, %sib5.exit ], [ %s6.next, %sib6.header ]
+ %s6.next = add i64 %s6, 1
+ %s6.ec = icmp eq i64 %s6.next, 4
+ br i1 %s6.ec, label %sib6.exit, label %sib6.header
+
+sib6.exit:
+ br label %sib7.header
+
+sib7.header:
+ %s7 = phi i64 [ 0, %sib6.exit ], [ %s7.next, %sib7.header ]
+ %s7.next = add i64 %s7, 1
+ %s7.ec = icmp eq i64 %s7.next, 4
+ br i1 %s7.ec, label %sib7.exit, label %sib7.header
+
+sib7.exit:
+ br label %sib8.header
+
+sib8.header:
+ %s8 = phi i64 [ 0, %sib7.exit ], [ %s8.next, %sib8.header ]
+ %s8.next = add i64 %s8, 1
+ %s8.ec = icmp eq i64 %s8.next, 4
+ br i1 %s8.ec, label %sib8.exit, label %sib8.header
+
+sib8.exit:
+ br label %sib9.header
+
+sib9.header:
+ %s9 = phi i64 [ 0, %sib8.exit ], [ %s9.next, %sib9.header ]
+ %s9.next = add i64 %s9, 1
+ %s9.ec = icmp eq i64 %s9.next, 4
+ br i1 %s9.ec, label %sib9.exit, label %sib9.header
+
+sib9.exit:
+ br label %top.latch
+
+top.latch:
+ %t.next = add i64 %t, 1
+ %t.ec = icmp eq i64 %t.next, 4
+ br i1 %t.ec, label %exit, label %top.header
+
+exit:
+ ret void
+}
+
+; The pairX reduction cycle and its address expression are unchanged, and the
+; inner reduction PHIs still take their initial value from %pairX.outer.header
+; (interchange would rewire these incoming edges). The pair remains nested
+; between %top.header and %top.latch, with the sibling chain intact.
+; IR-LABEL: define void @bfs_loop_count_is_not_depth(
+; IR: %t = phi i64 [ 0, %entry ], [ %t.next, %top.latch ]
+; IR: %sumA.i = phi double [ 0.000000e+00, %top.header ], [ %sumA.i.lcssa, %pairX.outer.latch ]
+; IR: %sumC.i = phi double [ 0.000000e+00, %top.header ], [ %sumC.i.lcssa, %pairX.outer.latch ]
+; IR: %j = phi i64 [ 0, %pairX.outer.header ], [ %j.next, %pairX.inner ]
+; IR: %sumA.j = phi double [ %sumA.i, %pairX.outer.header ], [ %sumA.j.next, %pairX.inner ]
+; IR: %sumC.j = phi double [ %sumC.i, %pairX.outer.header ], [ %sumC.j.next, %pairX.inner ]
+; IR: %idxA = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+; IR: %sumA.j.next = fadd reassoc double %sumA.j, %a
+; IR: %sumC.j.next = fadd reassoc double %sumC.j, %c
+; IR: %sumA.i.lcssa = phi double [ %sumA.j.next, %pairX.inner ]
+; IR: %sumA.live = phi double [ %sumA.i.lcssa, %pairX.outer.latch ]
+; IR: store double %sumA.live, ptr %R, align 8
+; IR: %s1 = phi i64 [ 0, %pairX.exit ], [ %s1.next, %sib1.header ]
+; IR: %s9 = phi i64 [ 0, %sib8.exit ], [ %s9.next, %sib9.header ]
+; IR: %t.next = add i64 %t, 1
+
+;-------------------------------------------------------------------------------
+; (2) A single SCEV-uncomputable sibling loop (data-dependent exit) sits beside
+; a separate, computable, profitable fixed-1335 pair under a common ancestor.
+; isComputableLoopNest rejects the whole flat list, so the pair is not reached
+; and no analysis remark is emitted.
+;-------------------------------------------------------------------------------
+define void @uncomputable_sibling_does_not_block(ptr %A, ptr %U, ptr %R) {
+entry:
+ br label %anc.header
+
+anc.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %anc.latch ]
+ br label %pair.outer.header
+
+pair.outer.header:
+ %i = phi i64 [ 0, %anc.header ], [ %i.next, %pair.outer.latch ]
+ %sum.i = phi double [ 0.000000e+00, %anc.header ], [ %sum.i.lcssa, %pair.outer.latch ]
+ br label %pair.inner
+
+pair.inner:
+ %j = phi i64 [ 0, %pair.outer.header ], [ %j.next, %pair.inner ]
+ %sum.j = phi double [ %sum.i, %pair.outer.header ], [ %sum.j.next, %pair.inner ]
+ %idx = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+ %a = load double, ptr %idx, align 8
+ %sum.j.next = fadd reassoc double %sum.j, %a
+ %j.next = add i64 %j, 1
+ %j.ec = icmp eq i64 %j.next, 1335
+ br i1 %j.ec, label %pair.outer.latch, label %pair.inner
+
+pair.outer.latch:
+ %sum.i.lcssa = phi double [ %sum.j.next, %pair.inner ]
+ %i.next = add i64 %i, 1
+ %i.ec = icmp eq i64 %i.next, 1335
+ br i1 %i.ec, label %pair.exit, label %pair.outer.header
+
+pair.exit:
+ %sum.live = phi double [ %sum.i.lcssa, %pair.outer.latch ]
+ store double %sum.live, ptr %R, align 8
+ br label %usib.header
+
+usib.header:
+ %s = phi i64 [ 0, %pair.exit ], [ %s.next, %usib.header ]
+ %sp = getelementptr inbounds double, ptr %U, i64 %s
+ %sv = load double, ptr %sp, align 8
+ %sc = fcmp oeq double %sv, 0.000000e+00
+ %s.next = add i64 %s, 1
+ br i1 %sc, label %usib.exit, label %usib.header
+
+usib.exit:
+ br label %anc.latch
+
+anc.latch:
+ %k.next = add i64 %k, 1
+ %k.ec = icmp eq i64 %k.next, 4
+ br i1 %k.ec, label %exit, label %anc.header
+
+exit:
+ ret void
+}
+
+; The computable pair keeps its original order; the uncomputable sibling still
+; exits on a loaded value.
+; IR-LABEL: define void @uncomputable_sibling_does_not_block(
+; IR: %sum.i = phi double [ 0.000000e+00, %anc.header ], [ %sum.i.lcssa, %pair.outer.latch ]
+; IR: %sum.j = phi double [ %sum.i, %pair.outer.header ], [ %sum.j.next, %pair.inner ]
+; IR: %idx = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+; IR: %sum.j.next = fadd reassoc double %sum.j, %a
+; IR: %sum.i.lcssa = phi double [ %sum.j.next, %pair.inner ]
+; IR: %sum.live = phi double [ %sum.i.lcssa, %pair.outer.latch ]
+; IR: store double %sum.live, ptr %R, align 8
+; IR: %sv = load double, ptr %sp, align 8
+; IR: %sc = fcmp oeq double %sv, 0.000000e+00
+
+;-------------------------------------------------------------------------------
+; (3) A lower, computable fixed-1335 pair beneath an uncomputable *true
+; ancestor* (data-dependent latch). The chain is linear, but isComputableLoopNest
+; rejects it because of the ancestor, so the lower pair is never partitioned off
+; and considered.
+;-------------------------------------------------------------------------------
+define void @uncomputable_ancestor_partition(ptr %A, ptr %U, ptr %R) {
+entry:
+ br label %anc.header
+
+anc.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %anc.latch ]
+ br label %pair.outer.header
+
+pair.outer.header:
+ %i = phi i64 [ 0, %anc.header ], [ %i.next, %pair.outer.latch ]
+ %sum.i = phi double [ 0.000000e+00, %anc.header ], [ %sum.i.lcssa, %pair.outer.latch ]
+ br label %pair.inner
+
+pair.inner:
+ %j = phi i64 [ 0, %pair.outer.header ], [ %j.next, %pair.inner ]
+ %sum.j = phi double [ %sum.i, %pair.outer.header ], [ %sum.j.next, %pair.inner ]
+ %idx = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+ %a = load double, ptr %idx, align 8
+ %sum.j.next = fadd reassoc double %sum.j, %a
+ %j.next = add i64 %j, 1
+ %j.ec = icmp eq i64 %j.next, 1335
+ br i1 %j.ec, label %pair.outer.latch, label %pair.inner
+
+pair.outer.latch:
+ %sum.i.lcssa = phi double [ %sum.j.next, %pair.inner ]
+ %i.next = add i64 %i, 1
+ %i.ec = icmp eq i64 %i.next, 1335
+ br i1 %i.ec, label %anc.latch, label %pair.outer.header
+
+anc.latch:
+ %sum.live = phi double [ %sum.i.lcssa, %pair.outer.latch ]
+ store double %sum.live, ptr %R, align 8
+ %kp = getelementptr inbounds double, ptr %U, i64 %k
+ %kv = load double, ptr %kp, align 8
+ %kc = fcmp oeq double %kv, 0.000000e+00
+ %k.next = add i64 %k, 1
+ br i1 %kc, label %exit, label %anc.header
+
+exit:
+ ret void
+}
+
+; The pair is untouched and still nested under the uncomputable ancestor.
+; IR-LABEL: define void @uncomputable_ancestor_partition(
+; IR: %k = phi i64 [ 0, %entry ], [ %k.next, %anc.latch ]
+; IR: %sum.i = phi double [ 0.000000e+00, %anc.header ], [ %sum.i.lcssa, %pair.outer.latch ]
+; IR: %sum.j = phi double [ %sum.i, %pair.outer.header ], [ %sum.j.next, %pair.inner ]
+; IR: %idx = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+; IR: %sum.j.next = fadd reassoc double %sum.j, %a
+; IR: %sum.i.lcssa = phi double [ %sum.j.next, %pair.inner ]
+; IR: %sum.live = phi double [ %sum.i.lcssa, %pair.outer.latch ]
+; IR: store double %sum.live, ptr %R, align 8
+; IR: %kv = load double, ptr %kp, align 8
+; IR: %kc = fcmp oeq double %kv, 0.000000e+00
+
+;-------------------------------------------------------------------------------
+; (4) Two eligible direct single-child pairs (pairA, pairB) share one ancestor,
+; making the breadth-first list non-linear. The current pass emits the analysis
+; remark, then bails at the linearity check, leaving both pairs unchanged. The
+; follow-on candidate-formation implementation, which transforms at most one
+; fallback candidate per nest, will interchange exactly one of them.
+;-------------------------------------------------------------------------------
+define void @two_candidate_pairs_one_fallback(ptr %A, ptr %B, ptr %R) {
+entry:
+ br label %anc.header
+
+anc.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %anc.latch ]
+ br label %pairA.outer.header
+
+pairA.outer.header:
+ %iA = phi i64 [ 0, %anc.header ], [ %iA.next, %pairA.outer.latch ]
+ %sumA.i = phi double [ 0.000000e+00, %anc.header ], [ %sumA.i.lcssa, %pairA.outer.latch ]
+ br label %pairA.inner
+
+pairA.inner:
+ %jA = phi i64 [ 0, %pairA.outer.header ], [ %jA.next, %pairA.inner ]
+ %sumA.j = phi double [ %sumA.i, %pairA.outer.header ], [ %sumA.j.next, %pairA.inner ]
+ %idxA = getelementptr inbounds [1335 x double], ptr %A, i64 %jA, i64 %iA
+ %a = load double, ptr %idxA, align 8
+ %sumA.j.next = fadd reassoc double %sumA.j, %a
+ %jA.next = add i64 %jA, 1
+ %jA.ec = icmp eq i64 %jA.next, 1335
+ br i1 %jA.ec, label %pairA.outer.latch, label %pairA.inner
+
+pairA.outer.latch:
+ %sumA.i.lcssa = phi double [ %sumA.j.next, %pairA.inner ]
+ %iA.next = add i64 %iA, 1
+ %iA.ec = icmp eq i64 %iA.next, 1335
+ br i1 %iA.ec, label %pairA.exit, label %pairA.outer.header
+
+pairA.exit:
+ %sumA.live = phi double [ %sumA.i.lcssa, %pairA.outer.latch ]
+ store double %sumA.live, ptr %R, align 8
+ br label %pairB.outer.header
+
+pairB.outer.header:
+ %iB = phi i64 [ 0, %pairA.exit ], [ %iB.next, %pairB.outer.latch ]
+ %sumB.i = phi double [ 0.000000e+00, %pairA.exit ], [ %sumB.i.lcssa, %pairB.outer.latch ]
+ br label %pairB.inner
+
+pairB.inner:
+ %jB = phi i64 [ 0, %pairB.outer.header ], [ %jB.next, %pairB.inner ]
+ %sumB.j = phi double [ %sumB.i, %pairB.outer.header ], [ %sumB.j.next, %pairB.inner ]
+ %idxB = getelementptr inbounds [1335 x double], ptr %B, i64 %jB, i64 %iB
+ %b = load double, ptr %idxB, align 8
+ %sumB.j.next = fadd reassoc double %sumB.j, %b
+ %jB.next = add i64 %jB, 1
+ %jB.ec = icmp eq i64 %jB.next, 1335
+ br i1 %jB.ec, label %pairB.outer.latch, label %pairB.inner
+
+pairB.outer.latch:
+ %sumB.i.lcssa = phi double [ %sumB.j.next, %pairB.inner ]
+ %iB.next = add i64 %iB, 1
+ %iB.ec = icmp eq i64 %iB.next, 1335
+ br i1 %iB.ec, label %pairB.exit, label %pairB.outer.header
+
+pairB.exit:
+ %sumB.live = phi double [ %sumB.i.lcssa, %pairB.outer.latch ]
+ %rB = getelementptr inbounds double, ptr %R, i64 1
+ store double %sumB.live, ptr %rB, align 8
+ br label %anc.latch
+
+anc.latch:
+ %k.next = add i64 %k, 1
+ %k.ec = icmp eq i64 %k.next, 4
+ br i1 %k.ec, label %exit, label %anc.header
+
+exit:
+ ret void
+}
+
+; Both pairs keep their original order and address expressions.
+; IR-LABEL: define void @two_candidate_pairs_one_fallback(
+; IR: %sumA.j = phi double [ %sumA.i, %pairA.outer.header ], [ %sumA.j.next, %pairA.inner ]
+; IR: %idxA = getelementptr inbounds [1335 x double], ptr %A, i64 %jA, i64 %iA
+; IR: %sumA.j.next = fadd reassoc double %sumA.j, %a
+; IR: %sumA.i.lcssa = phi double [ %sumA.j.next, %pairA.inner ]
+; IR: %sumA.live = phi double [ %sumA.i.lcssa, %pairA.outer.latch ]
+; IR: store double %sumA.live, ptr %R, align 8
+; IR: %sumB.j = phi double [ %sumB.i, %pairB.outer.header ], [ %sumB.j.next, %pairB.inner ]
+; IR: %idxB = getelementptr inbounds [1335 x double], ptr %B, i64 %jB, i64 %iB
+; IR: %sumB.j.next = fadd reassoc double %sumB.j, %b
+; IR: %sumB.i.lcssa = phi double [ %sumB.j.next, %pairB.inner ]
+; IR: %sumB.live = phi double [ %sumB.i.lcssa, %pairB.outer.latch ]
+; IR: store double %sumB.live, ptr %rB, align 8
+
+;-------------------------------------------------------------------------------
+; (5a) Lasting negative: a partly-exact reduction set. sumA is reassociable but
+; sumB is a strict fadd, so even though this is a plain admissible 2-deep nest
+; the current pass refuses it (UnsupportedPHIOuter). It must stay refused after
+; the follow-on candidate-formation implementation as well -- reassociation is
+; required on every reordered recurrence.
+;-------------------------------------------------------------------------------
+define void @partly_exact_reduction(ptr %A, ptr %B, ptr %R) {
+entry:
+ br label %outer.header
+
+outer.header:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %outer.latch ]
+ %sumA.i = phi double [ 0.000000e+00, %entry ], [ %sumA.i.lcssa, %outer.latch ]
+ %sumB.i = phi double [ 0.000000e+00, %entry ], [ %sumB.i.lcssa, %outer.latch ]
+ br label %inner.header
+
+inner.header:
+ %j = phi i64 [ 0, %outer.header ], [ %j.next, %inner.header ]
+ %sumA.j = phi double [ %sumA.i, %outer.header ], [ %sumA.j.next, %inner.header ]
+ %sumB.j = phi double [ %sumB.i, %outer.header ], [ %sumB.j.next, %inner.header ]
+ %idxA = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+ %a = load double, ptr %idxA, align 8
+ %sumA.j.next = fadd reassoc double %sumA.j, %a
+ %idxB = getelementptr inbounds [1335 x double], ptr %B, i64 %j, i64 %i
+ %b = load double, ptr %idxB, align 8
+ %sumB.j.next = fadd double %sumB.j, %b
+ %j.next = add i64 %j, 1
+ %j.ec = icmp eq i64 %j.next, 1335
+ br i1 %j.ec, label %outer.latch, label %inner.header
+
+outer.latch:
+ %sumA.i.lcssa = phi double [ %sumA.j.next, %inner.header ]
+ %sumB.i.lcssa = phi double [ %sumB.j.next, %inner.header ]
+ %i.next = add i64 %i, 1
+ %i.ec = icmp eq i64 %i.next, 1335
+ br i1 %i.ec, label %exit, label %outer.header
+
+exit:
+ %sumA.res = phi double [ %sumA.i.lcssa, %outer.latch ]
+ %sumB.res = phi double [ %sumB.i.lcssa, %outer.latch ]
+ %rB = getelementptr inbounds double, ptr %R, i64 1
+ store double %sumA.res, ptr %R, align 8
+ store double %sumB.res, ptr %rB, align 8
+ ret void
+}
+
+; Both reductions keep their original order; note the strict (non-reassoc) fadd.
+; IR-LABEL: define void @partly_exact_reduction(
+; IR: %sumA.j = phi double [ %sumA.i, %outer.header ], [ %sumA.j.next, %inner.header ]
+; IR: %sumB.j = phi double [ %sumB.i, %outer.header ], [ %sumB.j.next, %inner.header ]
+; IR: %sumA.j.next = fadd reassoc double %sumA.j, %a
+; IR: %sumB.j.next = fadd double %sumB.j, %b
+; IR: %sumA.i.lcssa = phi double [ %sumA.j.next, %inner.header ]
+; IR: %sumB.i.lcssa = phi double [ %sumB.j.next, %inner.header ]
+; IR: %sumA.res = phi double [ %sumA.i.lcssa, %outer.latch ]
+; IR: %sumB.res = phi double [ %sumB.i.lcssa, %outer.latch ]
+; IR: %rB = getelementptr inbounds double, ptr %R, i64 1
+; IR: store double %sumA.res, ptr %R, align 8
+; IR: store double %sumB.res, ptr %rB, align 8
+
+;-------------------------------------------------------------------------------
+; (5b) Lasting negative: a dynamic leading dimension (A[j*n + i]). Here it is
+; embedded beside a sibling loop so the current pass bails at the linearity
+; check regardless of profitability -- the point this precommit pins is simply
+; that it is not transformed. Once the follow-on candidate-formation
+; implementation lands, its fallback reaches this pair and default
+; profitability declines the dynamic stride, so it must remain out of scope.
+;-------------------------------------------------------------------------------
+define void @dynamic_leading_dimension_subnest(ptr %A, i64 %n, ptr %U, ptr %R) {
+entry:
+ br label %anc.header
+
+anc.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %anc.latch ]
+ br label %pair.outer.header
+
+pair.outer.header:
+ %i = phi i64 [ 0, %anc.header ], [ %i.next, %pair.outer.latch ]
+ %sum.i = phi double [ 0.000000e+00, %anc.header ], [ %sum.i.lcssa, %pair.outer.latch ]
+ br label %pair.inner
+
+pair.inner:
+ %j = phi i64 [ 0, %pair.outer.header ], [ %j.next, %pair.inner ]
+ %sum.j = phi double [ %sum.i, %pair.outer.header ], [ %sum.j.next, %pair.inner ]
+ %rowoff = mul i64 %j, %n
+ %off = add i64 %rowoff, %i
+ %idx = getelementptr inbounds double, ptr %A, i64 %off
+ %a = load double, ptr %idx, align 8
+ %sum.j.next = fadd reassoc double %sum.j, %a
+ %j.next = add i64 %j, 1
+ %j.ec = icmp eq i64 %j.next, 1335
+ br i1 %j.ec, label %pair.outer.latch, label %pair.inner
+
+pair.outer.latch:
+ %sum.i.lcssa = phi double [ %sum.j.next, %pair.inner ]
+ %i.next = add i64 %i, 1
+ %i.ec = icmp eq i64 %i.next, 1335
+ br i1 %i.ec, label %pair.exit, label %pair.outer.header
+
+pair.exit:
+ %sum.live = phi double [ %sum.i.lcssa, %pair.outer.latch ]
+ store double %sum.live, ptr %R, align 8
+ br label %sib.header
+
+sib.header:
+ %s = phi i64 [ 0, %pair.exit ], [ %s.next, %sib.header ]
+ %sp = getelementptr inbounds double, ptr %U, i64 %s
+ %sv = load double, ptr %sp, align 8
+ %sd = fadd reassoc double %sv, 1.000000e+00
+ store double %sd, ptr %sp, align 8
+ %s.next = add i64 %s, 1
+ %s.ec = icmp eq i64 %s.next, 4
+ br i1 %s.ec, label %sib.exit, label %sib.header
+
+sib.exit:
+ br label %anc.latch
+
+anc.latch:
+ %k.next = add i64 %k, 1
+ %k.ec = icmp eq i64 %k.next, 4
+ br i1 %k.ec, label %exit, label %anc.header
+
+exit:
+ ret void
+}
+
+; The dynamic-stride address and reduction order are preserved.
+; IR-LABEL: define void @dynamic_leading_dimension_subnest(
+; IR: %sum.j = phi double [ %sum.i, %pair.outer.header ], [ %sum.j.next, %pair.inner ]
+; IR: %rowoff = mul i64 %j, %n
+; IR: %off = add i64 %rowoff, %i
+; IR: %idx = getelementptr inbounds double, ptr %A, i64 %off
+; IR: %sum.j.next = fadd reassoc double %sum.j, %a
+; IR: %sum.i.lcssa = phi double [ %sum.j.next, %pair.inner ]
+; IR: %sum.live = phi double [ %sum.i.lcssa, %pair.outer.latch ]
+; IR: store double %sum.live, ptr %R, align 8
+; IR: %sd = fadd reassoc double %sv, 1.000000e+00
+
+;-------------------------------------------------------------------------------
+; (5c) Lasting negative inside a fallback-triggering shape: an all-exact (strict,
+; non-reassoc) reduction pair nested under an ancestor k beside a sibling loop, so
+; the flat list is non-linear and the current pass bails at the linearity check
+; after the analysis remark. The follow-on candidate-formation fallback reaches
+; this direct i/j pair but must decline it -- reordering a strict fadd changes
+; the result, so reassoc is required on every reordered recurrence. The result
+; is stored (real live-out).
+;-------------------------------------------------------------------------------
+define void @all_exact_reduction_subnest(ptr %A, ptr %U, ptr %R) {
+entry:
+ br label %anc.header
+
+anc.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %anc.latch ]
+ br label %pair.outer.header
+
+pair.outer.header:
+ %i = phi i64 [ 0, %anc.header ], [ %i.next, %pair.outer.latch ]
+ %sum.i = phi double [ 0.000000e+00, %anc.header ], [ %sum.i.lcssa, %pair.outer.latch ]
+ br label %pair.inner
+
+pair.inner:
+ %j = phi i64 [ 0, %pair.outer.header ], [ %j.next, %pair.inner ]
+ %sum.j = phi double [ %sum.i, %pair.outer.header ], [ %sum.j.next, %pair.inner ]
+ %idx = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+ %a = load double, ptr %idx, align 8
+ %sum.j.next = fadd double %sum.j, %a
+ %j.next = add i64 %j, 1
+ %j.ec = icmp eq i64 %j.next, 1335
+ br i1 %j.ec, label %pair.outer.latch, label %pair.inner
+
+pair.outer.latch:
+ %sum.i.lcssa = phi double [ %sum.j.next, %pair.inner ]
+ %i.next = add i64 %i, 1
+ %i.ec = icmp eq i64 %i.next, 1335
+ br i1 %i.ec, label %pair.exit, label %pair.outer.header
+
+pair.exit:
+ %sum.live = phi double [ %sum.i.lcssa, %pair.outer.latch ]
+ store double %sum.live, ptr %R, align 8
+ br label %sib.header
+
+sib.header:
+ %s = phi i64 [ 0, %pair.exit ], [ %s.next, %sib.header ]
+ %sp = getelementptr inbounds double, ptr %U, i64 %s
+ %sv = load double, ptr %sp, align 8
+ %sd = fadd reassoc double %sv, 1.000000e+00
+ store double %sd, ptr %sp, align 8
+ %s.next = add i64 %s, 1
+ %s.ec = icmp eq i64 %s.next, 4
+ br i1 %s.ec, label %sib.exit, label %sib.header
+
+sib.exit:
+ br label %anc.latch
+
+anc.latch:
+ %k.next = add i64 %k, 1
+ %k.ec = icmp eq i64 %k.next, 4
+ br i1 %k.ec, label %exit, label %anc.header
+
+exit:
+ ret void
+}
+
+; The strict (non-reassoc) reduction cycle and its address stay in original
+; order; the observable store and the sibling traffic remain in place.
+; IR-LABEL: define void @all_exact_reduction_subnest(
+; IR: %sum.i = phi double [ 0.000000e+00, %anc.header ], [ %sum.i.lcssa, %pair.outer.latch ]
+; IR: %sum.j = phi double [ %sum.i, %pair.outer.header ], [ %sum.j.next, %pair.inner ]
+; IR: %idx = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+; IR: %sum.j.next = fadd double %sum.j, %a
+; IR: %sum.i.lcssa = phi double [ %sum.j.next, %pair.inner ]
+; IR: %sum.live = phi double [ %sum.i.lcssa, %pair.outer.latch ]
+; IR: store double %sum.live, ptr %R, align 8
+
+;-------------------------------------------------------------------------------
+; (5d) Lasting negative inside the same fallback-triggering shape: a partly-exact
+; reduction pair (sumA reassociable, sumB a strict fadd) nested under ancestor k
+; beside a sibling, so the current pass again bails at the linearity check. The
+; follow-on candidate-formation fallback reaches the i/j pair but must decline
+; it -- every reordered recurrence must be reassociable and sumB is not. Both
+; results are stored.
+;-------------------------------------------------------------------------------
+define void @partly_exact_reduction_subnest(ptr %A, ptr %B, ptr %U, ptr %R) {
+entry:
+ br label %anc.header
+
+anc.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %anc.latch ]
+ br label %pair.outer.header
+
+pair.outer.header:
+ %i = phi i64 [ 0, %anc.header ], [ %i.next, %pair.outer.latch ]
+ %sumA.i = phi double [ 0.000000e+00, %anc.header ], [ %sumA.i.lcssa, %pair.outer.latch ]
+ %sumB.i = phi double [ 0.000000e+00, %anc.header ], [ %sumB.i.lcssa, %pair.outer.latch ]
+ br label %pair.inner
+
+pair.inner:
+ %j = phi i64 [ 0, %pair.outer.header ], [ %j.next, %pair.inner ]
+ %sumA.j = phi double [ %sumA.i, %pair.outer.header ], [ %sumA.j.next, %pair.inner ]
+ %sumB.j = phi double [ %sumB.i, %pair.outer.header ], [ %sumB.j.next, %pair.inner ]
+ %idxA = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+ %a = load double, ptr %idxA, align 8
+ %sumA.j.next = fadd reassoc double %sumA.j, %a
+ %idxB = getelementptr inbounds [1335 x double], ptr %B, i64 %j, i64 %i
+ %b = load double, ptr %idxB, align 8
+ %sumB.j.next = fadd double %sumB.j, %b
+ %j.next = add i64 %j, 1
+ %j.ec = icmp eq i64 %j.next, 1335
+ br i1 %j.ec, label %pair.outer.latch, label %pair.inner
+
+pair.outer.latch:
+ %sumA.i.lcssa = phi double [ %sumA.j.next, %pair.inner ]
+ %sumB.i.lcssa = phi double [ %sumB.j.next, %pair.inner ]
+ %i.next = add i64 %i, 1
+ %i.ec = icmp eq i64 %i.next, 1335
+ br i1 %i.ec, label %pair.exit, label %pair.outer.header
+
+pair.exit:
+ %sumA.live = phi double [ %sumA.i.lcssa, %pair.outer.latch ]
+ %sumB.live = phi double [ %sumB.i.lcssa, %pair.outer.latch ]
+ %rB = getelementptr inbounds double, ptr %R, i64 1
+ store double %sumA.live, ptr %R, align 8
+ store double %sumB.live, ptr %rB, align 8
+ br label %sib.header
+
+sib.header:
+ %s = phi i64 [ 0, %pair.exit ], [ %s.next, %sib.header ]
+ %sp = getelementptr inbounds double, ptr %U, i64 %s
+ %sv = load double, ptr %sp, align 8
+ %sd = fadd reassoc double %sv, 1.000000e+00
+ store double %sd, ptr %sp, align 8
+ %s.next = add i64 %s, 1
+ %s.ec = icmp eq i64 %s.next, 4
+ br i1 %s.ec, label %sib.exit, label %sib.header
+
+sib.exit:
+ br label %anc.latch
+
+anc.latch:
+ %k.next = add i64 %k, 1
+ %k.ec = icmp eq i64 %k.next, 4
+ br i1 %k.ec, label %exit, label %anc.header
+
+exit:
+ ret void
+}
+
+; sumA is reassociable and sumB is a strict fadd; both keep their original order,
+; both results are stored, and the sibling traffic remains.
+; IR-LABEL: define void @partly_exact_reduction_subnest(
+; IR: %sumA.j.next = fadd reassoc double %sumA.j, %a
+; IR: %sumB.j.next = fadd double %sumB.j, %b
+; IR: %sumA.i.lcssa = phi double [ %sumA.j.next, %pair.inner ]
+; IR: %sumB.i.lcssa = phi double [ %sumB.j.next, %pair.inner ]
+; IR: %sumA.live = phi double [ %sumA.i.lcssa, %pair.outer.latch ]
+; IR: %sumB.live = phi double [ %sumB.i.lcssa, %pair.outer.latch ]
+; IR: %rB = getelementptr inbounds double, ptr %R, i64 1
+; IR: store double %sumA.live, ptr %R, align 8
+; IR: store double %sumB.live, ptr %rB, align 8
+
+;-------------------------------------------------------------------------------
+; (5e) Lasting negative: an unsupported (non-leaf) candidate structure. The
+; eligible-looking i/j pair has an inner loop j that is itself NOT a leaf -- it
+; encloses two sibling loops m and n -- so j has two children and the flat list
+; is non-linear. The current pass emits the analysis remark and bails at the
+; linearity check. The follow-on candidate-formation fallback initially selects
+; only a leaf inner loop, so this non-leaf candidate must remain skipped. The
+; candidate store is observable.
+;-------------------------------------------------------------------------------
+define void @non_leaf_candidate_subnest(ptr %A) {
+entry:
+ br label %i.header
+
+i.header:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %i.latch ]
+ br label %j.header
+
+j.header:
+ %j = phi i64 [ 0, %i.header ], [ %j.next, %j.latch ]
+ %idx = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+ store double 1.000000e+00, ptr %idx, align 8
+ br label %m.header
+
+m.header:
+ %m = phi i64 [ 0, %j.header ], [ %m.next, %m.header ]
+ %m.next = add i64 %m, 1
+ %m.ec = icmp eq i64 %m.next, 4
+ br i1 %m.ec, label %m.exit, label %m.header
+
+m.exit:
+ br label %n.header
+
+n.header:
+ %n = phi i64 [ 0, %m.exit ], [ %n.next, %n.header ]
+ %n.next = add i64 %n, 1
+ %n.ec = icmp eq i64 %n.next, 4
+ br i1 %n.ec, label %j.latch, label %n.header
+
+j.latch:
+ %j.next = add i64 %j, 1
+ %j.ec = icmp eq i64 %j.next, 1335
+ br i1 %j.ec, label %i.latch, label %j.header
+
+i.latch:
+ %i.next = add i64 %i, 1
+ %i.ec = icmp eq i64 %i.next, 1335
+ br i1 %i.ec, label %exit, label %i.header
+
+exit:
+ ret void
+}
+
+; The i/j candidate keeps its original order and address, and the inner loop j
+; still encloses the two sibling loops m and n (its non-leaf body). The store is
+; the observable result.
+; IR-LABEL: define void @non_leaf_candidate_subnest(
+; IR: %i = phi i64 [ 0, %entry ], [ %i.next, %i.latch ]
+; IR: %j = phi i64 [ 0, %i.header ], [ %j.next, %j.latch ]
+; IR: %idx = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+; IR: store double 1.000000e+00, ptr %idx, align 8
+; IR: %m = phi i64 [ 0, %j.header ], [ %m.next, %m.header ]
+; IR: %n = phi i64 [ 0, %m.exit ], [ %n.next, %n.header ]
diff --git a/llvm/test/Transforms/LoopInterchange/inner-subnest-dependences.ll b/llvm/test/Transforms/LoopInterchange/inner-subnest-dependences.ll
new file mode 100644
index 0000000000000..e45e6266ab44a
--- /dev/null
+++ b/llvm/test/Transforms/LoopInterchange/inner-subnest-dependences.ll
@@ -0,0 +1,414 @@
+; Precommit test for the surrounding dependence context of an inner subnest
+; candidate in LoopInterchange.
+;
+; This file is entirely hand-maintained. Do NOT run update_test_checks.py on it.
+;
+; Every function has a true ancestor loop `k`, an adjacent candidate pair
+; `i`(outer)/`j`(inner), and a *sibling* loop `s` with its own memory traffic.
+; Because the ancestor has two child loops the breadth-first LoopNest list is
+; not a single linear chain. LoopInterchangePass::run still emits the analysis
+; remark (the depth and computability checks pass), and then
+; LoopInterchange::run(LoopNest&) bails at its linearity check -- nothing is
+; interchanged today. That is all this precommit pins: the analysis remark plus
+; the actual unswapped IR (original k/i/j nesting, address expressions, and the
+; sibling's separate memory operations left in place).
+;
+; The four access patterns set up the ancestor-prefix cases that the follow-on
+; candidate-formation implementation will distinguish (known-forward,
+; equal/legal, unknown, equal/unsafe), and the last function makes the sibling
+; store overlap the candidate array to pin that the follow-on implementation
+; must NOT fold sibling memory into the candidate's direction matrix. This
+; precommit deliberately does not assert those legality outcomes or direction
+; vectors; the unmodified pass never computes the candidate pair's matrix here.
+;
+; RUN: opt < %s -passes=loop-interchange -cache-line-size=64 \
+; RUN: -verify-dom-info -verify-loop-info -verify-scev -verify-loop-lcssa \
+; RUN: -S 2>&1 | FileCheck %s --check-prefix=IR
+;
+; RUN: opt < %s -passes=loop-interchange -cache-line-size=64 \
+; RUN: -pass-remarks=loop-interchange -pass-remarks-missed=loop-interchange \
+; RUN: -pass-remarks-output=%t -disable-output
+; RUN: FileCheck %s --check-prefix=YAML --input-file=%t
+;
+; The unknown-ancestor fixture must expose a *real*, non-confused surrounding
+; dependence whose ancestor (k) column is unknown while the selected i/j columns
+; are known and legal in isolation. Prove that on the unmodified pass with the
+; dependence-analysis printer: DA reports a genuine flow/anti dependence whose
+; outermost (k) direction is `*` -- not `confused!`. (LoopInterchange's own
+; matrix normalizes this exact CF[j-1][i] = CF[j][i] shape to `* = <`; see the
+; all_eq_lt case in legality-check.ll.)
+; RUN: opt < %s -passes='print<da>' -aa-pipeline=basic-aa -disable-output 2>&1 \
+; RUN: | FileCheck %s --check-prefix=DA
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+
+ at KF = global [8 x [8 x [8 x double]]] zeroinitializer
+ at EQ = global [8 x [8 x [8 x double]]] zeroinitializer
+ at US = global [8 x [8 x [8 x double]]] zeroinitializer
+ at CF = global [8 x [8 x double]] zeroinitializer
+ at SB = global [8 x [8 x double]] zeroinitializer
+
+; Each nest reaches the transform (analysis remark) but is then rejected as
+; non-linear; none is interchanged.
+; YAML: --- !Analysis
+; YAML: Name: Dependence
+; YAML: Function: dep_known_forward_ancestor
+; YAML: --- !Analysis
+; YAML: Name: Dependence
+; YAML: Function: dep_equal_legal_ancestor
+; YAML: --- !Analysis
+; YAML: Name: Dependence
+; YAML: Function: dep_unknown_ancestor
+; YAML: --- !Analysis
+; YAML: Name: Dependence
+; YAML: Function: dep_equal_unsafe_ancestor
+; YAML: --- !Analysis
+; YAML: Name: Dependence
+; YAML: Function: sibling_store_not_a_candidate_dimension
+
+;-------------------------------------------------------------------------------
+; Known-forward ancestor prefix: the store to KF[k+1][i][j] is read at KF[k][i][j]
+; on the next k iteration, a lexicographically forward carry on the ancestor.
+; That forward prefix is decisive for the follow-on candidate-formation
+; implementation, which will find the i/j swap legal; here the nest is simply
+; not processed (non-linear).
+;-------------------------------------------------------------------------------
+define void @dep_known_forward_ancestor() {
+entry:
+ br label %k.header
+
+k.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %k.latch ]
+ br label %i.header
+
+i.header:
+ %i = phi i64 [ 0, %k.header ], [ %i.next, %i.latch ]
+ br label %j.body
+
+j.body:
+ %j = phi i64 [ 0, %i.header ], [ %j.next, %j.body ]
+ %kp1 = add i64 %k, 1
+ %ld.idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @KF, i64 0, i64 %k, i64 %i, i64 %j
+ %v = load double, ptr %ld.idx, align 8
+ %nv = fadd double %v, 1.000000e+00
+ %st.idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @KF, i64 0, i64 %kp1, i64 %i, i64 %j
+ store double %nv, ptr %st.idx, align 8
+ %j.next = add i64 %j, 1
+ %j.ec = icmp eq i64 %j.next, 7
+ br i1 %j.ec, label %i.latch, label %j.body
+
+i.latch:
+ %i.next = add i64 %i, 1
+ %i.ec = icmp eq i64 %i.next, 7
+ br i1 %i.ec, label %sib.preheader, label %i.header
+
+sib.preheader:
+ br label %sib.header
+
+sib.header:
+ %s = phi i64 [ 0, %sib.preheader ], [ %s.next, %sib.header ]
+ %s.idx = getelementptr inbounds [8 x [8 x double]], ptr @SB, i64 0, i64 %k, i64 %s
+ %s.v = load double, ptr %s.idx, align 8
+ %s.nv = fadd double %s.v, 1.000000e+00
+ store double %s.nv, ptr %s.idx, align 8
+ %s.next = add i64 %s, 1
+ %s.ec = icmp eq i64 %s.next, 7
+ br i1 %s.ec, label %k.latch, label %sib.header
+
+k.latch:
+ %k.next = add i64 %k, 1
+ %k.ec = icmp eq i64 %k.next, 7
+ br i1 %k.ec, label %exit, label %k.header
+
+exit:
+ ret void
+}
+
+; Original k/i/j nesting and the KF[k]/KF[k+1] carry are preserved, and the
+; sibling still writes SB[k][s].
+; IR-LABEL: define void @dep_known_forward_ancestor(
+; IR: %k = phi i64 [ 0, %entry ], [ %k.next, %k.latch ]
+; IR: %i = phi i64 [ 0, %k.header ], [ %i.next, %i.latch ]
+; IR: %j = phi i64 [ 0, %i.header ], [ %j.next, %j.body ]
+; IR: %ld.idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @KF, i64 0, i64 %k, i64 %i, i64 %j
+; IR: %st.idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @KF, i64 0, i64 %kp1, i64 %i, i64 %j
+; IR: store double %nv, ptr %st.idx, align 8
+; IR: %s = phi i64 [ 0, %sib.preheader ], [ %s.next, %sib.header ]
+; IR: %s.idx = getelementptr inbounds [8 x [8 x double]], ptr @SB, i64 0, i64 %k, i64 %s
+
+;-------------------------------------------------------------------------------
+; Equal/legal ancestor prefix: a read-modify-write of the same EQ[k][i][j], a
+; loop-independent (equal) dependence. For the follow-on candidate-formation
+; implementation the equal ancestor prefix delegates to the i/j columns, which
+; are legal to swap.
+;-------------------------------------------------------------------------------
+define void @dep_equal_legal_ancestor() {
+entry:
+ br label %k.header
+
+k.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %k.latch ]
+ br label %i.header
+
+i.header:
+ %i = phi i64 [ 0, %k.header ], [ %i.next, %i.latch ]
+ br label %j.body
+
+j.body:
+ %j = phi i64 [ 0, %i.header ], [ %j.next, %j.body ]
+ %idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @EQ, i64 0, i64 %k, i64 %i, i64 %j
+ %v = load double, ptr %idx, align 8
+ %nv = fadd double %v, 1.000000e+00
+ store double %nv, ptr %idx, align 8
+ %j.next = add i64 %j, 1
+ %j.ec = icmp eq i64 %j.next, 7
+ br i1 %j.ec, label %i.latch, label %j.body
+
+i.latch:
+ %i.next = add i64 %i, 1
+ %i.ec = icmp eq i64 %i.next, 7
+ br i1 %i.ec, label %sib.preheader, label %i.header
+
+sib.preheader:
+ br label %sib.header
+
+sib.header:
+ %s = phi i64 [ 0, %sib.preheader ], [ %s.next, %sib.header ]
+ %s.idx = getelementptr inbounds [8 x [8 x double]], ptr @SB, i64 0, i64 %k, i64 %s
+ %s.v = load double, ptr %s.idx, align 8
+ %s.nv = fadd double %s.v, 1.000000e+00
+ store double %s.nv, ptr %s.idx, align 8
+ %s.next = add i64 %s, 1
+ %s.ec = icmp eq i64 %s.next, 7
+ br i1 %s.ec, label %k.latch, label %sib.header
+
+k.latch:
+ %k.next = add i64 %k, 1
+ %k.ec = icmp eq i64 %k.next, 7
+ br i1 %k.ec, label %exit, label %k.header
+
+exit:
+ ret void
+}
+
+; IR-LABEL: define void @dep_equal_legal_ancestor(
+; IR: %k = phi i64 [ 0, %entry ], [ %k.next, %k.latch ]
+; IR: %i = phi i64 [ 0, %k.header ], [ %i.next, %i.latch ]
+; IR: %j = phi i64 [ 0, %i.header ], [ %j.next, %j.body ]
+; IR: %idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @EQ, i64 0, i64 %k, i64 %i, i64 %j
+; IR: store double %nv, ptr %idx, align 8
+; IR: %s.idx = getelementptr inbounds [8 x [8 x double]], ptr @SB, i64 0, i64 %k, i64 %s
+
+;-------------------------------------------------------------------------------
+; Unknown ancestor prefix: the surrounding k loop does not index CF at all, so
+; the ancestor column of the candidate pair's dependence is unknown (`*`), while
+; the selected i/j columns are perfectly known -- equal in i and unit-carried in
+; j (CF[j-1][i] = CF[j][i], the all_eq_lt shape from legality-check.ll whose
+; interchange matrix is `* = <`). Dependence analysis returns a real dependence,
+; not `confused!` (see the DA run). The strict fallback in the follow-on
+; candidate-formation implementation must reject this pair because the true
+; ancestor context is unknown; an unsound implementation that dropped or
+; projected the ancestor away would see only the legal `[= <]` i/j columns and
+; wrongly accept. This precommit only pins that the dependence is real and that
+; nothing is interchanged.
+;-------------------------------------------------------------------------------
+define void @dep_unknown_ancestor() {
+entry:
+ br label %k.header
+
+k.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %k.latch ]
+ br label %i.header
+
+i.header:
+ %i = phi i64 [ 0, %k.header ], [ %i.next, %i.latch ]
+ br label %j.body
+
+j.body:
+ %j = phi i64 [ 1, %i.header ], [ %j.next, %j.body ]
+ %jm1 = sub i64 %j, 1
+ %ld.idx = getelementptr inbounds [8 x [8 x double]], ptr @CF, i64 0, i64 %j, i64 %i
+ %v = load double, ptr %ld.idx, align 8
+ %nv = fadd double %v, 1.000000e+00
+ %st.idx = getelementptr inbounds [8 x [8 x double]], ptr @CF, i64 0, i64 %jm1, i64 %i
+ store double %nv, ptr %st.idx, align 8
+ %j.next = add i64 %j, 1
+ %j.ec = icmp eq i64 %j.next, 8
+ br i1 %j.ec, label %i.latch, label %j.body
+
+i.latch:
+ %i.next = add i64 %i, 1
+ %i.ec = icmp eq i64 %i.next, 7
+ br i1 %i.ec, label %sib.preheader, label %i.header
+
+sib.preheader:
+ br label %sib.header
+
+sib.header:
+ %s = phi i64 [ 0, %sib.preheader ], [ %s.next, %sib.header ]
+ %s.idx = getelementptr inbounds [8 x [8 x double]], ptr @SB, i64 0, i64 %k, i64 %s
+ %s.v = load double, ptr %s.idx, align 8
+ %s.nv = fadd double %s.v, 1.000000e+00
+ store double %s.nv, ptr %s.idx, align 8
+ %s.next = add i64 %s, 1
+ %s.ec = icmp eq i64 %s.next, 7
+ br i1 %s.ec, label %k.latch, label %sib.header
+
+k.latch:
+ %k.next = add i64 %k, 1
+ %k.ec = icmp eq i64 %k.next, 7
+ br i1 %k.ec, label %exit, label %k.header
+
+exit:
+ ret void
+}
+
+; The ancestor k does not index CF, so the surrounding context is unknown; the
+; candidate reads CF[j][i] and writes CF[j-1][i] in original i/j order, and the
+; sibling still writes SB[k][s]. Nothing is interchanged.
+; IR-LABEL: define void @dep_unknown_ancestor(
+; IR: %k = phi i64 [ 0, %entry ], [ %k.next, %k.latch ]
+; IR: %i = phi i64 [ 0, %k.header ], [ %i.next, %i.latch ]
+; IR: %j = phi i64 [ 1, %i.header ], [ %j.next, %j.body ]
+; IR: %ld.idx = getelementptr inbounds [8 x [8 x double]], ptr @CF, i64 0, i64 %j, i64 %i
+; IR: %st.idx = getelementptr inbounds [8 x [8 x double]], ptr @CF, i64 0, i64 %jm1, i64 %i
+; IR: store double %nv, ptr %st.idx, align 8
+; IR: %s.idx = getelementptr inbounds [8 x [8 x double]], ptr @SB, i64 0, i64 %k, i64 %s
+
+; DA reports a genuine (non-confused) anti-dependence for the CF read/write pair.
+; The ancestor is scalar (`S`, mapped conservatively to `*` by
+; LoopInterchange), while the selected i/j levels have known distances 0/1.
+; The normalized interchange matrix is `* = <`.
+; DA-LABEL: 'dep_unknown_ancestor'
+; DA: da analyze - anti [S 0 1]!
+
+;-------------------------------------------------------------------------------
+; Equal ancestor prefix, unsafe selected columns: US[k][i][j+1] is written from
+; US[k][i+1][j], a cross i/j carry that is not safe to interchange even though
+; the ancestor prefix is equal. The i/j columns themselves must reject in the
+; follow-on candidate-formation implementation.
+;-------------------------------------------------------------------------------
+define void @dep_equal_unsafe_ancestor() {
+entry:
+ br label %k.header
+
+k.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %k.latch ]
+ br label %i.header
+
+i.header:
+ %i = phi i64 [ 0, %k.header ], [ %i.next, %i.latch ]
+ %ip1 = add i64 %i, 1
+ br label %j.body
+
+j.body:
+ %j = phi i64 [ 0, %i.header ], [ %j.next, %j.body ]
+ %jp1 = add i64 %j, 1
+ %ld.idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @US, i64 0, i64 %k, i64 %ip1, i64 %j
+ %v = load double, ptr %ld.idx, align 8
+ %nv = fadd double %v, 1.000000e+00
+ %st.idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @US, i64 0, i64 %k, i64 %i, i64 %jp1
+ store double %nv, ptr %st.idx, align 8
+ %j.next = add i64 %j, 1
+ %j.ec = icmp eq i64 %j.next, 7
+ br i1 %j.ec, label %i.latch, label %j.body
+
+i.latch:
+ %i.next = add i64 %i, 1
+ %i.ec = icmp eq i64 %i.next, 7
+ br i1 %i.ec, label %sib.preheader, label %i.header
+
+sib.preheader:
+ br label %sib.header
+
+sib.header:
+ %s = phi i64 [ 0, %sib.preheader ], [ %s.next, %sib.header ]
+ %s.idx = getelementptr inbounds [8 x [8 x double]], ptr @SB, i64 0, i64 %k, i64 %s
+ %s.v = load double, ptr %s.idx, align 8
+ %s.nv = fadd double %s.v, 1.000000e+00
+ store double %s.nv, ptr %s.idx, align 8
+ %s.next = add i64 %s, 1
+ %s.ec = icmp eq i64 %s.next, 7
+ br i1 %s.ec, label %k.latch, label %sib.header
+
+k.latch:
+ %k.next = add i64 %k, 1
+ %k.ec = icmp eq i64 %k.next, 7
+ br i1 %k.ec, label %exit, label %k.header
+
+exit:
+ ret void
+}
+
+; IR-LABEL: define void @dep_equal_unsafe_ancestor(
+; IR: %i = phi i64 [ 0, %k.header ], [ %i.next, %i.latch ]
+; IR: %j = phi i64 [ 0, %i.header ], [ %j.next, %j.body ]
+; IR: %ld.idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @US, i64 0, i64 %k, i64 %ip1, i64 %j
+; IR: %st.idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @US, i64 0, i64 %k, i64 %i, i64 %jp1
+; IR: store double %nv, ptr %st.idx, align 8
+
+;-------------------------------------------------------------------------------
+; The sibling loop stores into the *same* array KF that the candidate reads, but
+; on a different (diagonal) index. If the follow-on candidate-formation
+; implementation ever collected the candidate pair's direction matrix from the
+; whole ancestor subtree it would pull in this sibling store and mis-model it as
+; a candidate dimension. Today the nest is simply non-linear and untouched; this
+; precommit pins that both the candidate access and the sibling store are
+; present and unchanged.
+;-------------------------------------------------------------------------------
+define void @sibling_store_not_a_candidate_dimension() {
+entry:
+ br label %k.header
+
+k.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %k.latch ]
+ br label %i.header
+
+i.header:
+ %i = phi i64 [ 0, %k.header ], [ %i.next, %i.latch ]
+ br label %j.body
+
+j.body:
+ %j = phi i64 [ 0, %i.header ], [ %j.next, %j.body ]
+ %idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @KF, i64 0, i64 %k, i64 %i, i64 %j
+ %v = load double, ptr %idx, align 8
+ %nv = fadd double %v, 1.000000e+00
+ store double %nv, ptr %idx, align 8
+ %j.next = add i64 %j, 1
+ %j.ec = icmp eq i64 %j.next, 7
+ br i1 %j.ec, label %i.latch, label %j.body
+
+i.latch:
+ %i.next = add i64 %i, 1
+ %i.ec = icmp eq i64 %i.next, 7
+ br i1 %i.ec, label %sib.preheader, label %i.header
+
+sib.preheader:
+ br label %sib.header
+
+sib.header:
+ %s = phi i64 [ 0, %sib.preheader ], [ %s.next, %sib.header ]
+ %sib.idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @KF, i64 0, i64 %k, i64 %s, i64 %s
+ store double 1.000000e+00, ptr %sib.idx, align 8
+ %s.next = add i64 %s, 1
+ %s.ec = icmp eq i64 %s.next, 7
+ br i1 %s.ec, label %k.latch, label %sib.header
+
+k.latch:
+ %k.next = add i64 %k, 1
+ %k.ec = icmp eq i64 %k.next, 7
+ br i1 %k.ec, label %exit, label %k.header
+
+exit:
+ ret void
+}
+
+; The candidate KF[k][i][j] access and the sibling diagonal store KF[k][s][s]
+; both remain, with the pair in original order.
+; IR-LABEL: define void @sibling_store_not_a_candidate_dimension(
+; IR: %i = phi i64 [ 0, %k.header ], [ %i.next, %i.latch ]
+; IR: %j = phi i64 [ 0, %i.header ], [ %j.next, %j.body ]
+; IR: %idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @KF, i64 0, i64 %k, i64 %i, i64 %j
+; IR: store double %nv, ptr %idx, align 8
+; IR: %sib.idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @KF, i64 0, i64 %k, i64 %s, i64 %s
+; IR: store double 1.000000e+00, ptr %sib.idx, align 8
>From 4a6f6018a6eb7f23b20337a2694fcbf0d58f22a8 Mon Sep 17 00:00:00 2001
From: "Matt P. Dziubinski" <matt-p.dziubinski at hpe.com>
Date: Fri, 7 Aug 2026 02:44:34 -0500
Subject: [PATCH 3/3] [LoopInterchange] Consider eligible inner subnests
LoopNest::getLoops() is a breadth-first descendant list, not a nesting
depth. Use getNestDepth() for depth policy so shallow sibling-rich nests
are not rejected as too deep.
When the complete nest is unsuitable for the standard multi-swap path,
consider direct parent and leaf-child pairs deepest first. Carry depth through
one breadth-first walk, cap retained candidates and attempts, preserve the
full ancestor dependence prefix, exclude sibling traffic, preserve LoopInfo
sibling order, and transform at most one legal, profitable pair.
Scope cache analysis to each candidate root-to-leaf ancestor chain. Non-linear
nests route directly to fallback and no longer emit the standard-path Computed
remark. The hidden switch disables fallback transformation, not the corrected
depth policy or dispatch diagnostics.
Add exact coverage for depth, dependence, ordering, budgets, cache
profitability, complexity bounds, and disabled behavior.
Assisted-by: Claude Opus 4.8, Claude Opus 5, Claude Sonnet 5, GPT-5.6 Sol.
---
.../lib/Transforms/Scalar/LoopInterchange.cpp | 362 ++++++++-
.../LoopInterchange/inner-subnest-budget.ll | 710 ++++++++++++++++++
.../inner-subnest-cache-cost.ll | 372 +++++++++
.../inner-subnest-candidates.ll | 661 ++++++++++++----
.../inner-subnest-dependences.ll | 233 ++++--
.../inner-subnest-enumeration.ll | 315 ++++++++
.../inner-subnest-fallback-switch.ll | 84 +++
.../LoopInterchange/large-nested-6d.ll | 36 +-
8 files changed, 2511 insertions(+), 262 deletions(-)
create mode 100644 llvm/test/Transforms/LoopInterchange/inner-subnest-budget.ll
create mode 100644 llvm/test/Transforms/LoopInterchange/inner-subnest-cache-cost.ll
create mode 100644 llvm/test/Transforms/LoopInterchange/inner-subnest-enumeration.ll
create mode 100644 llvm/test/Transforms/LoopInterchange/inner-subnest-fallback-switch.ll
diff --git a/llvm/lib/Transforms/Scalar/LoopInterchange.cpp b/llvm/lib/Transforms/Scalar/LoopInterchange.cpp
index e667decf7149c..fadc83d378d15 100644
--- a/llvm/lib/Transforms/Scalar/LoopInterchange.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopInterchange.cpp
@@ -100,6 +100,15 @@ static cl::opt<unsigned int> MaxLoopNestDepth(
"loop-interchange-max-loop-nest-depth", cl::init(10), cl::Hidden,
cl::desc("Maximum depth of loop nest considered for the transform"));
+static cl::opt<unsigned int> MaxInnerSubnestCandidates(
+ "loop-interchange-max-inner-subnest-candidates", cl::init(10), cl::Hidden,
+ cl::desc("Maximum number of inner-subnest fallback candidates attempted"));
+
+static cl::opt<bool> EnableInnerSubnestFallback(
+ "loop-interchange-enable-inner-subnest-fallback", cl::init(true),
+ cl::Hidden,
+ cl::desc("Enable fallback interchange of eligible inner subnests"));
+
// We prefer cache cost to vectorization by default.
static cl::list<RuleTy> Profitabilities(
"loop-interchange-profitabilities", cl::MiscFlags::CommaSeparated,
@@ -399,6 +408,23 @@ static bool isLegalToInterChangeLoops(CharMatrix &DepMatrix,
return true;
}
+// For an inner-subnest fallback candidate, the columns strictly outside the
+// selected pair (indices [0, OuterLoopId)) describe the surrounding ancestor
+// context. The fallback accepts a pair only when every row's prefix is either
+// lexicographically forward -- a '<' before any '>'/'*', which is decisive --
+// or entirely equal/independent, which delegates to the candidate columns. A
+// prefix whose leftmost non-equal/independent direction is unknown ('*') or
+// backward ('>') is rejected; an earlier '<' makes later columns irrelevant.
+// This is a conservative fallback policy, not a soundness requirement. The
+// standard multi-swap path accepts cases such as '* = <' and is unchanged.
+static bool hasDecisiveOrEqualAncestorPrefix(const CharMatrix &DepMatrix,
+ unsigned OuterLoopId) {
+ for (const std::vector<char> &Row : DepMatrix)
+ if (isLexicographicallyPositive(Row, 0, OuterLoopId) == false)
+ return false;
+ return true;
+}
+
static void populateWorklist(Loop &L, LoopVector &LoopList) {
LLVM_DEBUG(dbgs() << "Calling populateWorklist on Func: "
<< L.getHeader()->getParent()->getName() << " Loop: %"
@@ -422,22 +448,22 @@ static void populateWorklist(Loop &L, LoopVector &LoopList) {
LoopList.push_back(CurrentLoop);
}
-static bool hasSupportedLoopDepth(ArrayRef<Loop *> LoopList,
- OptimizationRemarkEmitter &ORE) {
- unsigned LoopNestDepth = LoopList.size();
- if (LoopNestDepth < MinLoopNestDepth || LoopNestDepth > MaxLoopNestDepth) {
- LLVM_DEBUG(dbgs() << "Unsupported depth of loop nest " << LoopNestDepth
+// Check the *true* nesting depth of the loop nest against the supported range.
+// \p NestDepth is LoopNest::getNestDepth(), i.e. the depth of the deepest loop,
+// not the breadth-first descendant count LoopNest::getLoops().size(), which
+// also counts sibling loops and is therefore not a nesting depth. \p
+// DescendantCount is reported only for diagnostics.
+//
+// The caller emits the missed remark only after any fallback attempt fails.
+static bool hasSupportedLoopDepth(unsigned NestDepth,
+ unsigned DescendantCount) {
+ if (NestDepth < MinLoopNestDepth || NestDepth > MaxLoopNestDepth) {
+ LLVM_DEBUG(dbgs() << "Unsupported depth of loop nest " << NestDepth
<< ", the supported range is [" << MinLoopNestDepth
<< ", " << MaxLoopNestDepth << "].\n");
- Loop *OuterLoop = LoopList.front();
- ORE.emit([&]() {
- return OptimizationRemarkMissed(DEBUG_TYPE, "UnsupportedLoopNestDepth",
- OuterLoop->getStartLoc(),
- OuterLoop->getHeader())
- << "Unsupported depth of loop nest, the supported range is ["
- << std::to_string(MinLoopNestDepth) << ", "
- << std::to_string(MaxLoopNestDepth) << "].\n";
- });
+ LLVM_DEBUG(dbgs() << " true nesting depth = " << NestDepth
+ << ", breadth-first descendant count = "
+ << DescendantCount << "\n");
return false;
}
return true;
@@ -463,6 +489,17 @@ static bool isComputableLoopNest(ScalarEvolution *SE,
return true;
}
+// The breadth-first LoopNest list is a single linear chain when each loop is
+// the parent of the next. Only such a list is a standard multi-swap candidate;
+// a non-linear list (a loop with sibling subloops) is handled by the
+// inner-subnest fallback instead of being silently dropped.
+static bool isLinearLoopList(ArrayRef<Loop *> LoopList) {
+ for (unsigned I = 1; I < LoopList.size(); ++I)
+ if (LoopList[I]->getParentLoop() != LoopList[I - 1])
+ return false;
+ return true;
+}
+
namespace {
/// LoopInterchangeLegality checks if it is legal to interchange the loop.
@@ -581,11 +618,12 @@ class LoopInterchangeLegality {
/// actually needed.
class CacheCostManager {
Loop *OutermostLoop;
+ std::optional<LoopVectorTy> ExplicitLoopNest;
LoopStandardAnalysisResults *AR;
DependenceInfo *DI;
- /// CacheCost for \ref OutermostLoop. Once it is computed, it is cached. Note
- /// that the result can be nullptr.
+ /// CacheCost for \ref OutermostLoop or \ref ExplicitLoopNest. Once it is
+ /// computed, it is cached. Note that the result can be nullptr.
std::optional<std::unique_ptr<CacheCost>> CC;
/// Maps each loop to an index representing the optimal position within the
@@ -598,6 +636,18 @@ class CacheCostManager {
CacheCostManager(Loop *OutermostLoop, LoopStandardAnalysisResults *AR,
DependenceInfo *DI)
: OutermostLoop(OutermostLoop), AR(AR), DI(DI) {}
+ CacheCostManager(ArrayRef<Loop *> LoopNest, LoopStandardAnalysisResults *AR,
+ DependenceInfo *DI)
+ : OutermostLoop(LoopNest.empty() ? nullptr : LoopNest.front()),
+ ExplicitLoopNest(LoopVectorTy(LoopNest.begin(), LoopNest.end())),
+ AR(AR), DI(DI) {
+ assert(!ExplicitLoopNest->empty() &&
+ "Explicit cache-cost nest must be non-empty");
+ for (unsigned I = 1; I < ExplicitLoopNest->size(); ++I)
+ assert((*ExplicitLoopNest)[I]->getParentLoop() ==
+ (*ExplicitLoopNest)[I - 1] &&
+ "Explicit cache-cost nest must be a parent-child chain");
+ }
CacheCost *getCacheCost();
const DenseMap<const Loop *, unsigned> &getCostMap();
};
@@ -666,6 +716,16 @@ class LoopInterchangeTransform {
const LoopInterchangeLegality &LIL;
};
+/// A direct parent/child loop pair considered by the inner-subnest fallback:
+/// \p Outer has exactly one child \p Inner, and \p Inner is a leaf loop.
+struct InnerSubnestCandidate {
+ Loop *Outer;
+ Loop *Inner;
+ /// Absolute depth of \p Inner, carried by the enumeration walk. This is also
+ /// the length of the Root..Inner chain because Root is outermost.
+ unsigned Depth;
+};
+
struct LoopInterchange {
ScalarEvolution *SE = nullptr;
LoopInfo *LI = nullptr;
@@ -691,9 +751,8 @@ struct LoopInterchange {
bool run(LoopNest &LN) {
SmallVector<Loop *, 8> LoopList(LN.getLoops());
- for (unsigned I = 1; I < LoopList.size(); ++I)
- if (LoopList[I]->getParentLoop() != LoopList[I - 1])
- return false;
+ assert(isLinearLoopList(LoopList) &&
+ "Standard interchange path expects a linear nest");
return processLoopList(LoopList);
}
@@ -706,8 +765,9 @@ struct LoopInterchange {
bool processLoopList(SmallVectorImpl<Loop *> &LoopList) {
bool Changed = false;
- // Ensure proper loop nest depth.
- assert(hasSupportedLoopDepth(LoopList, *ORE) &&
+ // Ensure proper loop nest depth. On this standard path the breadth-first
+ // list is a single linear chain, so its size equals the nesting depth.
+ assert(hasSupportedLoopDepth(LoopList.size(), LoopList.size()) &&
"Unsupported depth of loop nest.");
unsigned LoopNestDepth = LoopList.size();
@@ -815,6 +875,179 @@ struct LoopInterchange {
return true;
}
+
+ /// When the complete breadth-first LoopNest is unsuitable for the standard
+ /// path -- because it is non-linear, a loop in its linear chain is
+ /// uncomputable or unsupported, or it exceeds the depth policy -- consider
+ /// one sound, profitable adjacent inner pair. Enumerate direct single-child
+ /// parent/child edges whose inner loop is a leaf, try them deepest-first, and
+ /// perform at most one interchange through this fallback. The standard
+ /// multi-swap path and its behavior are unchanged.
+ bool tryInnerSubnestFallback(LoopNest &LN) {
+ Loop &Root = LN.getOutermostLoop();
+ assert(!Root.getParentLoop() &&
+ "Fallback root is expected to be an outermost loop");
+ if (!EnableInnerSubnestFallback)
+ return false;
+
+ LLVM_DEBUG(
+ dbgs() << "Considering inner-subnest fallback for loop nest '"
+ << Root.getName()
+ << "': breadth-first descendant count = " << LN.getNumLoops()
+ << ", true nesting depth = " << LN.getNestDepth() << "\n");
+
+ // Enumerate candidate pairs: Outer has exactly one child Inner and Inner is
+ // a leaf loop. Carry depth through this breadth-first walk instead of
+ // repeatedly walking parent chains. Do not filter with
+ // getPerfectLoops/arePerfectlyNested, which reject reduction nests that
+ // this pass's own legality accepts. Retain only the globally deepest
+ // MaxInnerSubnestCandidates candidates while scanning once: the vector is
+ // kept sorted by inner-loop depth (deepest first), and because a new
+ // candidate is inserted after all equal-depth entries, ties keep a stable
+ // breadth-first order. One extra slot is kept so that, when more eligible
+ // candidates exist than the budget allows, the deepest unattempted pair
+ // can anchor a stable budget-exhaustion remark. This bounds work to
+ // O(descendants * (MaxInnerSubnestCandidates + 1)) and auxiliary storage
+ // to O(descendants).
+ const unsigned Budget = MaxInnerSubnestCandidates;
+ SmallVector<InnerSubnestCandidate, 8> Candidates;
+ SmallVector<std::pair<Loop *, unsigned>, 16> Worklist;
+ Worklist.push_back({&Root, 1});
+ for (unsigned I = 0; I != Worklist.size(); ++I) {
+ auto [Outer, OuterDepth] = Worklist[I];
+ ArrayRef<Loop *> SubLoops = Outer->getSubLoops();
+ for (Loop *Child : SubLoops)
+ Worklist.push_back({Child, OuterDepth + 1});
+ if (SubLoops.size() != 1)
+ continue;
+ Loop *Inner = SubLoops.front();
+ if (!Inner->isInnermost())
+ continue;
+ unsigned Depth = OuterDepth + 1;
+ if (Depth < MinLoopNestDepth || Depth > MaxLoopNestDepth)
+ continue;
+ auto Pos = Candidates.begin();
+ while (Pos != Candidates.end() && Pos->Depth >= Depth)
+ ++Pos;
+ Candidates.insert(Pos, {Outer, Inner, Depth});
+ // Keep at most Budget pairs to attempt, plus one overflow marker; drop
+ // the shallowest surplus. Compare the surplus instead of Budget + 1
+ // because the unsigned addition wraps when Budget is UINT_MAX.
+ if (Candidates.size() > Budget && Candidates.size() - Budget > 1)
+ Candidates.pop_back();
+ }
+
+ // Attempt at most Budget pairs, deepest first. Any retained pair beyond the
+ // budget is kept only to anchor the budget-exhaustion remark.
+ bool BudgetExhausted = Candidates.size() > Budget;
+ for (unsigned Idx = 0; Idx < Candidates.size() && Idx < Budget; ++Idx) {
+ const InnerSubnestCandidate &Cand = Candidates[Idx];
+
+ // Require pair-local canonical form, a computable trip count, a single
+ // backedge, and a unique exit for both loops of the pair.
+ SmallVector<Loop *, 2> Pair = {Cand.Outer, Cand.Inner};
+ if (!Cand.Outer->isLoopSimplifyForm() ||
+ !Cand.Inner->isLoopSimplifyForm() ||
+ !isComputableLoopNest(SE, Pair)) {
+ LLVM_DEBUG(dbgs() << "Inner-subnest candidate is not a supported, "
+ "computable inner pair.\n");
+ ORE->emit([&]() {
+ return OptimizationRemarkMissed(DEBUG_TYPE, "FallbackUnsupportedPair",
+ Cand.Inner->getStartLoc(),
+ Cand.Inner->getHeader())
+ << "Inner-subnest candidate is not a supported, computable "
+ "inner pair.";
+ });
+ continue;
+ }
+
+ // Build the ancestor chain from the true LoopNest root down to Inner.
+ // Because Inner is a leaf and Outer is its only parent, the selected
+ // Outer/Inner are the two innermost (adjacent) columns of that chain.
+ SmallVector<Loop *, 8> ChainInnerFirst;
+ ChainInnerFirst.reserve(Cand.Depth);
+ for (Loop *L = Cand.Inner;; L = L->getParentLoop()) {
+ ChainInnerFirst.push_back(L);
+ if (L == &Root)
+ break;
+ }
+ SmallVector<Loop *, 8> Ancestors(ChainInnerFirst.rbegin(),
+ ChainInnerFirst.rend());
+ unsigned InnerLoopId = Ancestors.size() - 1;
+ unsigned OuterLoopId = InnerLoopId - 1;
+ assert(Ancestors.size() == Cand.Depth && Ancestors.front() == &Root &&
+ Ancestors[InnerLoopId] == Cand.Inner &&
+ Ancestors[OuterLoopId] == Cand.Outer &&
+ "Unexpected inner-subnest ancestor chain");
+
+ // Build the dependence direction matrix over the *full* ancestor chain.
+ // Collect memory only from the candidate Outer's subtree so that
+ // breadth-first sibling loops are neither dependence inputs nor matrix
+ // columns. DependenceInfo reports absolute loop-depth levels. Because the
+ // LoopNest pass root has depth 1, level N maps to ancestor column N - 1.
+ // Deeper non-common levels are padded with independence as on the
+ // standard path.
+ CharMatrix DependencyMatrix;
+ if (!populateDependencyMatrix(DependencyMatrix, Ancestors.size(),
+ Cand.Outer, DI, SE, ORE)) {
+ LLVM_DEBUG(
+ dbgs() << "Populating inner-subnest dependency matrix failed.\n");
+ continue;
+ }
+ LLVM_DEBUG(
+ dbgs() << "Inner-subnest dependency matrix (ancestor chain):\n";
+ printDepMatrix(DependencyMatrix));
+
+ // Conservative surrounding-context gate: a known-forward ancestor prefix
+ // is decisive, while an all-equal/independent prefix delegates to the
+ // candidate columns. A leading unknown/confused/backward direction
+ // rejects.
+ if (!hasDecisiveOrEqualAncestorPrefix(DependencyMatrix, OuterLoopId)) {
+ LLVM_DEBUG(dbgs() << "Unknown or unsafe surrounding dependence context "
+ "for inner-subnest candidate.\n");
+ ORE->emit([&]() {
+ return OptimizationRemarkMissed(DEBUG_TYPE, "FallbackUnknownContext",
+ Cand.Inner->getStartLoc(),
+ Cand.Inner->getHeader())
+ << "Cannot interchange inner subnest: the surrounding "
+ "dependence context is unknown or unsafe.";
+ });
+ continue;
+ }
+
+ LLVM_DEBUG(dbgs() << "Selected inner-subnest candidate: Outer '"
+ << Cand.Outer->getName() << "' (loop id " << OuterLoopId
+ << ") and Inner '" << Cand.Inner->getName()
+ << "' (loop id " << InnerLoopId << ").\n");
+
+ // Reuse the existing per-pair legality, default profitability, transform,
+ // and analysis updates. processLoop swaps only the selected absolute
+ // ancestor columns. Build lazy cache analysis from that same linear chain
+ // so its reference groups come from the candidate leaf and exclude
+ // disjoint siblings. Perform at most one interchange through the
+ // fallback.
+ CacheCostManager CCM(Ancestors, AR, DI);
+ if (processLoop(Ancestors, InnerLoopId, OuterLoopId, DependencyMatrix,
+ CCM))
+ return true;
+ }
+
+ // Every attempted candidate failed. If eligible candidates remained beyond
+ // the attempt budget, emit a stable missed remark and leave the IR
+ // unchanged. This point is never reached after a successful transform,
+ // which returns above, so the remark cannot follow an applied interchange.
+ if (BudgetExhausted) {
+ const InnerSubnestCandidate &Overflow = Candidates.back();
+ ORE->emit([&]() {
+ return OptimizationRemarkMissed(DEBUG_TYPE, "FallbackCandidateBudget",
+ Overflow.Inner->getStartLoc(),
+ Overflow.Inner->getHeader())
+ << "Inner-subnest candidate budget exhausted; the loop nest is "
+ "left unchanged.";
+ });
+ }
+ return false;
+ }
};
} // end anonymous namespace
@@ -1538,7 +1771,8 @@ static bool areOuterLoopExitPHIsSupported(Loop *OuterLoop, Loop *InnerLoop) {
continue;
// The incoming value is defined in the outer loop latch. Currently we
- // only support that in case the outer loop latch has a single predecessor.
+ // only support that in case the outer loop latch has a single
+ // predecessor.
// This guarantees that the outer loop latch is executed if and only if
// the inner loop is executed (because tightlyNested() guarantees that the
// outer loop header only branches to the inner loop or the outer loop
@@ -1764,7 +1998,13 @@ void CacheCostManager::computeIfUnitinialized() {
return;
LLVM_DEBUG(dbgs() << "Compute CacheCost.\n");
- CC = CacheCost::getCacheCost(*OutermostLoop, *AR, *DI);
+ if (!ExplicitLoopNest)
+ CC = CacheCost::getCacheCost(*OutermostLoop, *AR, *DI);
+ else if (ExplicitLoopNest->empty())
+ CC = nullptr;
+ else
+ CC = std::make_unique<CacheCost>(*ExplicitLoopNest, AR->LI, AR->SE, AR->TTI,
+ AR->AA, *DI);
// Obtain the loop vector returned from loop cache analysis beforehand,
// and put each <Loop, index> pair into a map for constant time query
// later. Indices in loop vector reprsent the optimal order of the
@@ -2085,10 +2325,17 @@ void LoopInterchangeTransform::restructureLoops(
// Switch the loop levels.
if (OuterLoopParent) {
- // Remove the loop from its parent loop.
- removeChildLoop(OuterLoopParent, NewInner);
+ // Detach the new outer loop (original inner) from the new inner loop
+ // (original outer), then replace the new inner loop with the new outer
+ // loop *in place* in the parent's subloop list. Replacing in place --
+ // rather than removing the old child and appending the replacement --
+ // preserves the parent's sibling program order in LoopInfo. That order is
+ // observable (e.g. by a following loop-nest pass) and matters when the
+ // interchanged pair is nested beneath a parent with other sibling loops, as
+ // in the inner-subnest fallback. On the standard multi-swap path the parent
+ // has a single child, so this matches the previous remove/append.
removeChildLoop(NewInner, NewOuter);
- OuterLoopParent->addChildLoop(NewOuter);
+ OuterLoopParent->replaceChildLoopWith(NewInner, NewOuter);
} else {
removeChildLoop(NewInner, NewOuter);
LI->changeTopLevelLoop(NewInner, NewOuter);
@@ -2688,24 +2935,57 @@ PreservedAnalyses LoopInterchangePass::run(LoopNest &LN,
OptimizationRemarkEmitter ORE(&F);
- // Ensure minimum depth of the loop nest to do the interchange.
- if (!hasSupportedLoopDepth(LoopList, ORE))
- return PreservedAnalyses::all();
- // Ensure computable loop nest.
- if (!isComputableLoopNest(&AR.SE, LoopList)) {
+ // LoopNest::getLoops() is a breadth-first walk over *all* descendant loops
+ // (siblings included), so its size is the descendant count, not the nesting
+ // depth. Use the true nesting depth for the depth policy; keep the descendant
+ // count for diagnostics only.
+ unsigned DescendantCount = LoopList.size();
+ unsigned NestDepth = LN.getNestDepth();
+
+ DependenceInfo DI(&F, &AR.AA, &AR.SE, &AR.LI);
+ LoopInterchange Interchange(&AR.SE, &AR.LI, &DI, &AR.DT, &AR, &ORE);
+
+ bool Changed = false;
+ if (!hasSupportedLoopDepth(NestDepth, DescendantCount)) {
+ // The whole nest is outside the supported depth range. A genuinely too-deep
+ // nest may still contain an eligible inner pair; a too-shallow one cannot.
+ if (NestDepth > MaxLoopNestDepth)
+ Changed = Interchange.tryInnerSubnestFallback(LN);
+ // A successful fallback handles the nest despite its total depth.
+ if (!Changed) {
+ ORE.emit([&]() {
+ return OptimizationRemarkMissed(DEBUG_TYPE, "UnsupportedLoopNestDepth",
+ LN.getOutermostLoop().getStartLoc(),
+ LN.getOutermostLoop().getHeader())
+ << "Unsupported depth of loop nest, the supported range is ["
+ << std::to_string(MinLoopNestDepth) << ", "
+ << std::to_string(MaxLoopNestDepth) << "].\n";
+ });
+ }
+ } else if (!isLinearLoopList(LoopList)) {
+ // A non-linear nest always takes the fallback, so avoid whole-list SCEV
+ // work that cannot change the dispatch decision.
+ Changed = Interchange.tryInnerSubnestFallback(LN);
+ } else if (!isComputableLoopNest(&AR.SE, LoopList)) {
LLVM_DEBUG(dbgs() << "Not valid loop candidate for interchange\n");
- return PreservedAnalyses::all();
- }
+ // A loop in the linear chain has an uncomputable backedge count or an
+ // unsupported backedge/exit structure. Try the fallback on eligible,
+ // computable inner pairs.
+ Changed = Interchange.tryInnerSubnestFallback(LN);
+ } else {
+ ORE.emit([&]() {
+ return OptimizationRemarkAnalysis(DEBUG_TYPE, "Dependence",
+ LN.getOutermostLoop().getStartLoc(),
+ LN.getOutermostLoop().getHeader())
+ << "Computed dependence info, invoking the transform.";
+ });
- ORE.emit([&]() {
- return OptimizationRemarkAnalysis(DEBUG_TYPE, "Dependence",
- LN.getOutermostLoop().getStartLoc(),
- LN.getOutermostLoop().getHeader())
- << "Computed dependence info, invoking the transform.";
- });
+ // The caller dispatches non-linear nests to the fallback above, so run()
+ // handles only the standard multi-swap path for a linear chain.
+ Changed = Interchange.run(LN);
+ }
- DependenceInfo DI(&F, &AR.AA, &AR.SE, &AR.LI);
- if (!LoopInterchange(&AR.SE, &AR.LI, &DI, &AR.DT, &AR, &ORE).run(LN))
+ if (!Changed)
return PreservedAnalyses::all();
U.markLoopNestChanged(true);
return getLoopPassPreservedAnalyses();
diff --git a/llvm/test/Transforms/LoopInterchange/inner-subnest-budget.ll b/llvm/test/Transforms/LoopInterchange/inner-subnest-budget.ll
new file mode 100644
index 0000000000000..a01e155198c41
--- /dev/null
+++ b/llvm/test/Transforms/LoopInterchange/inner-subnest-budget.ll
@@ -0,0 +1,710 @@
+; Behavior tests for independent inner-subnest depth and candidate-count
+; policies. This file is hand-maintained; do NOT run update_test_checks.py.
+;
+; A maximum depth of 2 leaves the depth-3 nest unchanged. Separately, a maximum
+; of two candidate attempts in a supported depth-3 forest leaves all three
+; unknown-context pairs unchanged and emits a stable budget-exhaustion remark.
+; A companion positive fixture shows that a whole nest that is genuinely too
+; deep, but that contains a shallower eligible pair, is interchanged with no
+; contradictory "unsupported depth" remark. A separate one-loop fixture checks
+; the below-minimum diagnostic. A final fixture with more eligible candidates
+; than the budget allows, at four distinct depths, exercises the
+; candidate-retention eviction path.
+;
+; RUN: llvm-extract -S -func=over_cap_depth_filters_inner_pair %s \
+; RUN: -o %t.over-cap
+; RUN: opt < %t.over-cap -passes=loop-interchange -cache-line-size=64 \
+; RUN: -loop-interchange-max-loop-nest-depth=2 \
+; RUN: -verify-dom-info -verify-loop-info -verify-scev -verify-loop-lcssa \
+; RUN: -pass-remarks-output=%t.over-cap.yaml -disable-output
+; RUN: FileCheck %s --check-prefix=OVER-CAP \
+; RUN: --input-file=%t.over-cap.yaml --implicit-check-not=Interchanged
+; RUN: opt < %t.over-cap -passes='loop(loop-interchange),print<loops>' \
+; RUN: -cache-line-size=64 -loop-interchange-max-loop-nest-depth=2 \
+; RUN: -disable-output 2>&1 | FileCheck %s --check-prefix=OVER-CAP-LOOPS
+;
+; RUN: llvm-extract -S -func=budget_exhausted_all_fail %s -o %t.budget
+; RUN: opt < %t.budget -passes=loop-interchange -cache-line-size=64 \
+; RUN: -loop-interchange-max-loop-nest-depth=3 \
+; RUN: -loop-interchange-max-inner-subnest-candidates=2 \
+; RUN: -verify-dom-info -verify-loop-info -verify-scev -verify-loop-lcssa \
+; RUN: -pass-remarks-output=%t.budget.yaml -disable-output
+; RUN: FileCheck %s --check-prefix=BUDGET --input-file=%t.budget.yaml \
+; RUN: --implicit-check-not=Interchanged
+; RUN: opt < %t.budget -passes='loop(loop-interchange),print<loops>' \
+; RUN: -cache-line-size=64 -loop-interchange-max-loop-nest-depth=3 \
+; RUN: -loop-interchange-max-inner-subnest-candidates=2 \
+; RUN: -disable-output 2>&1 | FileCheck %s --check-prefix=BUDGET-LOOPS
+;
+; RUN: llvm-extract -S -func=too_deep_reaches_shallow_pair %s -o %t.too-deep
+; RUN: opt < %t.too-deep -passes=loop-interchange -cache-line-size=64 \
+; RUN: -loop-interchange-max-loop-nest-depth=3 \
+; RUN: -loop-interchange-profitabilities=ignore \
+; RUN: -verify-dom-info -verify-loop-info -verify-scev -verify-loop-lcssa \
+; RUN: -pass-remarks-output=%t.too-deep.yaml -disable-output
+; RUN: FileCheck %s --check-prefix=TOO-DEEP --input-file=%t.too-deep.yaml \
+; RUN: --implicit-check-not=Dependence \
+; RUN: --implicit-check-not=UnsupportedLoopNestDepth
+; RUN: opt < %t.too-deep -passes='loop(loop-interchange),print<loops>' \
+; RUN: -cache-line-size=64 -loop-interchange-max-loop-nest-depth=3 \
+; RUN: -loop-interchange-profitabilities=ignore \
+; RUN: -disable-output 2>&1 | FileCheck %s --check-prefix=TOO-DEEP-LOOPS
+; RUN: opt < %t.too-deep -passes=loop-interchange -cache-line-size=64 \
+; RUN: -loop-interchange-max-loop-nest-depth=3 \
+; RUN: -loop-interchange-enable-inner-subnest-fallback=false \
+; RUN: -loop-interchange-profitabilities=ignore \
+; RUN: -pass-remarks-output=%t.fallback-disabled.yaml -disable-output
+; RUN: FileCheck %s --check-prefix=FALLBACK-DISABLED \
+; RUN: --input-file=%t.fallback-disabled.yaml \
+; RUN: --implicit-check-not=Interchanged --implicit-check-not=Fallback
+; RUN: opt < %t.too-deep -passes='loop(loop-interchange),print<loops>' \
+; RUN: -cache-line-size=64 -loop-interchange-max-loop-nest-depth=3 \
+; RUN: -loop-interchange-enable-inner-subnest-fallback=false \
+; RUN: -loop-interchange-profitabilities=ignore \
+; RUN: -disable-output 2>&1 | \
+; RUN: FileCheck %s --check-prefix=FALLBACK-DISABLED-LOOPS
+;
+; RUN: llvm-extract -S -func=too_shallow_emits_depth_remark %s \
+; RUN: -o %t.too-shallow
+; RUN: opt < %t.too-shallow -passes=loop-interchange \
+; RUN: -pass-remarks-output=%t.too-shallow.yaml -disable-output
+; RUN: FileCheck %s --check-prefix=TOO-SHALLOW \
+; RUN: --input-file=%t.too-shallow.yaml
+;
+; RUN: llvm-extract -S -func=eviction_prunes_shallowest_surplus %s -o %t.evict
+; RUN: opt < %t.evict -passes=loop-interchange -cache-line-size=64 \
+; RUN: -loop-interchange-max-inner-subnest-candidates=2 \
+; RUN: -verify-dom-info -verify-loop-info -verify-scev -verify-loop-lcssa \
+; RUN: -pass-remarks-output=%t.evict.yaml -disable-output
+; RUN: FileCheck %s --check-prefix=EVICT --input-file=%t.evict.yaml \
+; RUN: --implicit-check-not=Interchanged
+; RUN: opt < %t.evict -passes='loop(loop-interchange),print<loops>' \
+; RUN: -cache-line-size=64 -loop-interchange-max-inner-subnest-candidates=2 \
+; RUN: -disable-output 2>&1 | FileCheck %s --check-prefix=EVICT-LOOPS
+; RUN: opt < %t.evict -passes=loop-interchange -cache-line-size=64 \
+; RUN: -loop-interchange-max-loop-nest-depth=5 \
+; RUN: -loop-interchange-max-inner-subnest-candidates=2 \
+; RUN: -verify-dom-info -verify-loop-info -verify-scev -verify-loop-lcssa \
+; RUN: -pass-remarks-output=%t.too-deep-fail.yaml -disable-output
+; RUN: FileCheck %s --check-prefix=TOO-DEEP-FAIL \
+; RUN: --input-file=%t.too-deep-fail.yaml \
+; RUN: --implicit-check-not=Interchanged
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+
+ at CF1 = global [8 x [8 x double]] zeroinitializer
+ at CF2 = global [8 x [8 x double]] zeroinitializer
+ at CF3 = global [8 x [8 x double]] zeroinitializer
+ at A = global [8 x [8 x [8 x double]]] zeroinitializer
+ at B = global [8 x [8 x [8 x double]]] zeroinitializer
+ at DA = global [8 x [8 x double]] zeroinitializer
+ at DB = global [8 x [8 x double]] zeroinitializer
+ at DC = global [8 x [8 x double]] zeroinitializer
+ at DD = global [8 x [8 x double]] zeroinitializer
+
+; OVER-CAP: --- !Missed
+; OVER-CAP: Name: UnsupportedLoopNestDepth
+; OVER-CAP-NEXT: Function: over_cap_depth_filters_inner_pair
+;
+; BUDGET-NOT: --- !
+; BUDGET: --- !Missed
+; BUDGET-NEXT: Pass: loop-interchange
+; BUDGET-NEXT: Name: FallbackUnknownContext
+; BUDGET-NEXT: DebugLoc: { File: inner-subnest-budget.ll, Line: 70, Column: 1 }
+; BUDGET-NEXT: Function: budget_exhausted_all_fail
+; BUDGET-NEXT: Args:
+; BUDGET-NEXT: - String: 'Cannot interchange inner subnest: the surrounding dependence context is unknown or unsafe.'
+; BUDGET-NEXT: ...
+; BUDGET-NEXT: --- !Missed
+; BUDGET-NEXT: Pass: loop-interchange
+; BUDGET-NEXT: Name: FallbackUnknownContext
+; BUDGET-NEXT: DebugLoc: { File: inner-subnest-budget.ll, Line: 80, Column: 1 }
+; BUDGET-NEXT: Function: budget_exhausted_all_fail
+; BUDGET-NEXT: Args:
+; BUDGET-NEXT: - String: 'Cannot interchange inner subnest: the surrounding dependence context is unknown or unsafe.'
+; BUDGET-NEXT: ...
+; BUDGET-NEXT: --- !Missed
+; BUDGET-NEXT: Pass: loop-interchange
+; BUDGET-NEXT: Name: FallbackCandidateBudget
+; BUDGET-NEXT: DebugLoc: { File: inner-subnest-budget.ll, Line: 90, Column: 1 }
+; BUDGET-NEXT: Function: budget_exhausted_all_fail
+; BUDGET-NEXT: Args:
+; BUDGET-NEXT: - String: 'Inner-subnest candidate budget exhausted; the loop nest is left unchanged.'
+; BUDGET-NEXT: ...
+; BUDGET-NOT: --- !
+;
+; The whole nest is too deep, but no "unsupported depth" remark appears because
+; the shallower pair is found and interchanged; only a Passed record fires.
+; The implicit exclusions also reject a supported-depth analysis record.
+; TOO-DEEP-NOT: --- !
+; TOO-DEEP: --- !Passed
+; TOO-DEEP-NEXT: Pass: loop-interchange
+; TOO-DEEP-NEXT: Name: Interchanged
+; TOO-DEEP-NEXT: Function: too_deep_reaches_shallow_pair
+; TOO-DEEP-NEXT: Args:
+; TOO-DEEP-NEXT: - String: Loop interchanged with enclosing loop.
+; TOO-DEEP-NEXT: ...
+; TOO-DEEP-NOT: --- !
+;
+; FALLBACK-DISABLED-NOT: --- !
+; FALLBACK-DISABLED: --- !Missed
+; FALLBACK-DISABLED-NEXT: Pass: loop-interchange
+; FALLBACK-DISABLED-NEXT: Name: UnsupportedLoopNestDepth
+; FALLBACK-DISABLED-NEXT: Function: too_deep_reaches_shallow_pair
+; FALLBACK-DISABLED-NEXT: Args:
+; FALLBACK-DISABLED-NEXT: - String: 'Unsupported depth of loop nest, the supported range is ['
+; FALLBACK-DISABLED-NEXT: - String: '2'
+; FALLBACK-DISABLED-NEXT: - String: ', '
+; FALLBACK-DISABLED-NEXT: - String: '3'
+; FALLBACK-DISABLED-NEXT: - String: "].\n"
+; FALLBACK-DISABLED-NEXT: ...
+; FALLBACK-DISABLED-NOT: --- !
+;
+; TOO-SHALLOW-NOT: --- !
+; TOO-SHALLOW: --- !Missed
+; TOO-SHALLOW-NEXT: Pass: loop-interchange
+; TOO-SHALLOW-NEXT: Name: UnsupportedLoopNestDepth
+; TOO-SHALLOW-NEXT: Function: too_shallow_emits_depth_remark
+; TOO-SHALLOW-NEXT: Args:
+; TOO-SHALLOW-NEXT: - String: 'Unsupported depth of loop nest, the supported range is ['
+; TOO-SHALLOW-NEXT: - String: '2'
+; TOO-SHALLOW-NEXT: - String: ', '
+; TOO-SHALLOW-NEXT: - String: '10'
+; TOO-SHALLOW-NEXT: - String: "].\n"
+; TOO-SHALLOW-NEXT: ...
+; TOO-SHALLOW-NOT: --- !
+;
+; Deepest-first: the two deepest retained candidates (depth 6, then depth 5)
+; are attempted and rejected; the depth-4 candidate is retained only to anchor
+; the overflow remark and is never attempted. The depth-3 candidate is evicted.
+; Distinct debug locations identify each candidate and exact NEXT checks reject
+; additional attempts.
+; EVICT-NOT: --- !
+; EVICT: --- !Missed
+; EVICT-NEXT: Pass: loop-interchange
+; EVICT-NEXT: Name: FallbackUnknownContext
+; EVICT-NEXT: DebugLoc: { File: inner-subnest-budget.ll, Line: 60, Column: 1 }
+; EVICT-NEXT: Function: eviction_prunes_shallowest_surplus
+; EVICT-NEXT: Args:
+; EVICT-NEXT: - String: 'Cannot interchange inner subnest: the surrounding dependence context is unknown or unsafe.'
+; EVICT-NEXT: ...
+; EVICT-NEXT: --- !Missed
+; EVICT-NEXT: Pass: loop-interchange
+; EVICT-NEXT: Name: FallbackUnknownContext
+; EVICT-NEXT: DebugLoc: { File: inner-subnest-budget.ll, Line: 50, Column: 1 }
+; EVICT-NEXT: Function: eviction_prunes_shallowest_surplus
+; EVICT-NEXT: Args:
+; EVICT-NEXT: - String: 'Cannot interchange inner subnest: the surrounding dependence context is unknown or unsafe.'
+; EVICT-NEXT: ...
+; EVICT-NEXT: --- !Missed
+; EVICT-NEXT: Pass: loop-interchange
+; EVICT-NEXT: Name: FallbackCandidateBudget
+; EVICT-NEXT: DebugLoc: { File: inner-subnest-budget.ll, Line: 40, Column: 1 }
+; EVICT-NEXT: Function: eviction_prunes_shallowest_surplus
+; EVICT-NEXT: Args:
+; EVICT-NEXT: - String: 'Inner-subnest candidate budget exhausted; the loop nest is left unchanged.'
+; EVICT-NEXT: ...
+; EVICT-NOT: --- !
+;
+; With maximum depth 5, the depth-6 candidate is filtered during enumeration.
+; The depth-5 and depth-4 candidates are attempted and rejected, the depth-3
+; candidate anchors the budget remark, and the final record reports the
+; unsupported whole-nest depth.
+; TOO-DEEP-FAIL-NOT: --- !
+; TOO-DEEP-FAIL: --- !Missed
+; TOO-DEEP-FAIL-NEXT: Pass: loop-interchange
+; TOO-DEEP-FAIL-NEXT: Name: FallbackUnknownContext
+; TOO-DEEP-FAIL-NEXT: DebugLoc: { File: inner-subnest-budget.ll, Line: 50, Column: 1 }
+; TOO-DEEP-FAIL-NEXT: Function: eviction_prunes_shallowest_surplus
+; TOO-DEEP-FAIL-NEXT: Args:
+; TOO-DEEP-FAIL-NEXT: - String: 'Cannot interchange inner subnest: the surrounding dependence context is unknown or unsafe.'
+; TOO-DEEP-FAIL-NEXT: ...
+; TOO-DEEP-FAIL-NEXT: --- !Missed
+; TOO-DEEP-FAIL-NEXT: Pass: loop-interchange
+; TOO-DEEP-FAIL-NEXT: Name: FallbackUnknownContext
+; TOO-DEEP-FAIL-NEXT: DebugLoc: { File: inner-subnest-budget.ll, Line: 40, Column: 1 }
+; TOO-DEEP-FAIL-NEXT: Function: eviction_prunes_shallowest_surplus
+; TOO-DEEP-FAIL-NEXT: Args:
+; TOO-DEEP-FAIL-NEXT: - String: 'Cannot interchange inner subnest: the surrounding dependence context is unknown or unsafe.'
+; TOO-DEEP-FAIL-NEXT: ...
+; TOO-DEEP-FAIL-NEXT: --- !Missed
+; TOO-DEEP-FAIL-NEXT: Pass: loop-interchange
+; TOO-DEEP-FAIL-NEXT: Name: FallbackCandidateBudget
+; TOO-DEEP-FAIL-NEXT: DebugLoc: { File: inner-subnest-budget.ll, Line: 30, Column: 1 }
+; TOO-DEEP-FAIL-NEXT: Function: eviction_prunes_shallowest_surplus
+; TOO-DEEP-FAIL-NEXT: Args:
+; TOO-DEEP-FAIL-NEXT: - String: 'Inner-subnest candidate budget exhausted; the loop nest is left unchanged.'
+; TOO-DEEP-FAIL-NEXT: ...
+; TOO-DEEP-FAIL-NEXT: --- !Missed
+; TOO-DEEP-FAIL-NEXT: Pass: loop-interchange
+; TOO-DEEP-FAIL-NEXT: Name: UnsupportedLoopNestDepth
+; TOO-DEEP-FAIL-NEXT: Function: eviction_prunes_shallowest_surplus
+; TOO-DEEP-FAIL-NEXT: Args:
+; TOO-DEEP-FAIL-NEXT: - String: 'Unsupported depth of loop nest, the supported range is ['
+; TOO-DEEP-FAIL-NEXT: - String: '2'
+; TOO-DEEP-FAIL-NEXT: - String: ', '
+; TOO-DEEP-FAIL-NEXT: - String: '5'
+; TOO-DEEP-FAIL-NEXT: - String: "].\n"
+; TOO-DEEP-FAIL-NEXT: ...
+; TOO-DEEP-FAIL-NOT: --- !
+
+;-------------------------------------------------------------------------------
+; The one-loop nest is below the supported minimum and has no fallback path.
+; The pass must retain its user-visible unsupported-depth remark.
+;-------------------------------------------------------------------------------
+define void @too_shallow_emits_depth_remark(ptr %A) {
+entry:
+ br label %loop
+
+loop:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %loop ]
+ %p = getelementptr inbounds double, ptr %A, i64 %i
+ store double 1.000000e+00, ptr %p, align 8
+ %i.next = add i64 %i, 1
+ %done = icmp eq i64 %i.next, 8
+ br i1 %done, label %exit, label %loop
+
+exit:
+ ret void
+}
+
+;-------------------------------------------------------------------------------
+; A linear, computable depth-3 nest. The cap (2) rejects the whole nest and its
+; depth-3 candidate pair, so no fallback interchange is attempted.
+;-------------------------------------------------------------------------------
+define void @over_cap_depth_filters_inner_pair(ptr %A, ptr %R) {
+entry:
+ br label %anc.header
+
+anc.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %anc.latch ]
+ br label %pair.outer.header
+
+pair.outer.header:
+ %i = phi i64 [ 0, %anc.header ], [ %i.next, %pair.outer.latch ]
+ %sum.i = phi double [ 0.000000e+00, %anc.header ], [ %sum.i.lcssa, %pair.outer.latch ]
+ br label %pair.inner
+
+pair.inner:
+ %j = phi i64 [ 0, %pair.outer.header ], [ %j.next, %pair.inner ]
+ %sum.j = phi double [ %sum.i, %pair.outer.header ], [ %sum.j.next, %pair.inner ]
+ %idx = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+ %a = load double, ptr %idx, align 8
+ %sum.j.next = fadd reassoc double %sum.j, %a
+ %j.next = add i64 %j, 1
+ %j.ec = icmp eq i64 %j.next, 1335
+ br i1 %j.ec, label %pair.outer.latch, label %pair.inner
+
+pair.outer.latch:
+ %sum.i.lcssa = phi double [ %sum.j.next, %pair.inner ]
+ %i.next = add i64 %i, 1
+ %i.ec = icmp eq i64 %i.next, 1335
+ br i1 %i.ec, label %anc.latch, label %pair.outer.header
+
+anc.latch:
+ %sum.live = phi double [ %sum.i.lcssa, %pair.outer.latch ]
+ store double %sum.live, ptr %R, align 8
+ %k.next = add i64 %k, 1
+ %k.ec = icmp eq i64 %k.next, 4
+ br i1 %k.ec, label %exit, label %anc.header
+
+exit:
+ ret void
+}
+
+; The over-cap pair remains in its original order.
+; OVER-CAP-LOOPS-LABEL: Loop info for function 'over_cap_depth_filters_inner_pair':
+; OVER-CAP-LOOPS: Loop at depth 1 containing: %anc.header<header>
+; OVER-CAP-LOOPS-NEXT: Loop at depth 2 containing: %pair.outer.header<header>
+; OVER-CAP-LOOPS-NEXT: Loop at depth 3 containing: %pair.inner<header>
+
+; Budget exhaustion leaves all three candidate pairs unswapped and in their
+; original sibling order: each pN.i header remains outer to its pN.j body.
+; BUDGET-LOOPS-LABEL: Loop info for function 'budget_exhausted_all_fail':
+; BUDGET-LOOPS: Loop at depth 1 containing: %k.header<header>
+; BUDGET-LOOPS-NEXT: Loop at depth 2 containing: %p1.i.header<header>
+; BUDGET-LOOPS-NEXT: Loop at depth 3 containing: %p1.j.body<header>
+; BUDGET-LOOPS-NEXT: Loop at depth 2 containing: %p2.i.header<header>
+; BUDGET-LOOPS-NEXT: Loop at depth 3 containing: %p2.j.body<header>
+; BUDGET-LOOPS-NEXT: Loop at depth 2 containing: %p3.i.header<header>
+; BUDGET-LOOPS-NEXT: Loop at depth 3 containing: %p3.j.body<header>
+
+;-------------------------------------------------------------------------------
+; Three eligible sibling pairs under a common ancestor k that does not index the
+; arrays, so each pair's surrounding (k) dependence direction is unknown (`*`).
+; Every pair is therefore rejected (FallbackUnknownContext). With the budget at
+; 2, only two are attempted deepest-first; because a third eligible pair remained
+; and all attempts failed, the budget-exhaustion remark fires.
+;-------------------------------------------------------------------------------
+define void @budget_exhausted_all_fail() !dbg !10 {
+entry:
+ br label %k.header
+
+k.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %k.latch ]
+ br label %p1.i.header
+
+p1.i.header:
+ %i1 = phi i64 [ 0, %k.header ], [ %i1.next, %p1.i.latch ]
+ br label %p1.j.body, !dbg !11
+
+p1.j.body:
+ %j1 = phi i64 [ 1, %p1.i.header ], [ %j1.next, %p1.j.body ]
+ %j1m1 = sub i64 %j1, 1
+ %p1.ld = getelementptr inbounds [8 x [8 x double]], ptr @CF1, i64 0, i64 %j1, i64 %i1
+ %p1.v = load double, ptr %p1.ld, align 8
+ %p1.nv = fadd double %p1.v, 1.000000e+00
+ %p1.st = getelementptr inbounds [8 x [8 x double]], ptr @CF1, i64 0, i64 %j1m1, i64 %i1
+ store double %p1.nv, ptr %p1.st, align 8
+ %j1.next = add i64 %j1, 1
+ %j1.ec = icmp eq i64 %j1.next, 8
+ br i1 %j1.ec, label %p1.i.latch, label %p1.j.body
+
+p1.i.latch:
+ %i1.next = add i64 %i1, 1
+ %i1.ec = icmp eq i64 %i1.next, 7
+ br i1 %i1.ec, label %p2.i.header, label %p1.i.header
+
+p2.i.header:
+ %i2 = phi i64 [ 0, %p1.i.latch ], [ %i2.next, %p2.i.latch ]
+ br label %p2.j.body, !dbg !12
+
+p2.j.body:
+ %j2 = phi i64 [ 1, %p2.i.header ], [ %j2.next, %p2.j.body ]
+ %j2m1 = sub i64 %j2, 1
+ %p2.ld = getelementptr inbounds [8 x [8 x double]], ptr @CF2, i64 0, i64 %j2, i64 %i2
+ %p2.v = load double, ptr %p2.ld, align 8
+ %p2.nv = fadd double %p2.v, 1.000000e+00
+ %p2.st = getelementptr inbounds [8 x [8 x double]], ptr @CF2, i64 0, i64 %j2m1, i64 %i2
+ store double %p2.nv, ptr %p2.st, align 8
+ %j2.next = add i64 %j2, 1
+ %j2.ec = icmp eq i64 %j2.next, 8
+ br i1 %j2.ec, label %p2.i.latch, label %p2.j.body
+
+p2.i.latch:
+ %i2.next = add i64 %i2, 1
+ %i2.ec = icmp eq i64 %i2.next, 7
+ br i1 %i2.ec, label %p3.i.header, label %p2.i.header
+
+p3.i.header:
+ %i3 = phi i64 [ 0, %p2.i.latch ], [ %i3.next, %p3.i.latch ]
+ br label %p3.j.body, !dbg !13
+
+p3.j.body:
+ %j3 = phi i64 [ 1, %p3.i.header ], [ %j3.next, %p3.j.body ]
+ %j3m1 = sub i64 %j3, 1
+ %p3.ld = getelementptr inbounds [8 x [8 x double]], ptr @CF3, i64 0, i64 %j3, i64 %i3
+ %p3.v = load double, ptr %p3.ld, align 8
+ %p3.nv = fadd double %p3.v, 1.000000e+00
+ %p3.st = getelementptr inbounds [8 x [8 x double]], ptr @CF3, i64 0, i64 %j3m1, i64 %i3
+ store double %p3.nv, ptr %p3.st, align 8
+ %j3.next = add i64 %j3, 1
+ %j3.ec = icmp eq i64 %j3.next, 8
+ br i1 %j3.ec, label %p3.i.latch, label %p3.j.body
+
+p3.i.latch:
+ %i3.next = add i64 %i3, 1
+ %i3.ec = icmp eq i64 %i3.next, 7
+ br i1 %i3.ec, label %k.latch, label %p3.i.header
+
+k.latch:
+ %k.next = add i64 %k, 1
+ %k.ec = icmp eq i64 %k.next, 7
+ br i1 %k.ec, label %exit, label %k.header
+
+exit:
+ ret void
+}
+
+;-------------------------------------------------------------------------------
+; A whole nest that is genuinely too deep (true depth 4 > cap 3) has two
+; independently-nested subtrees under a common depth-1 ancestor hk: hi/bx at
+; true depth 3, and ha/hb/bc at true depth 4. Only the depth-3 hi/bx pair is
+; within the cap and is interchanged through the fallback; the deeper ha/hb/bc
+; triple has no eligible parent/leaf-child pair within the cap (its only
+; candidate, hb/bc, is itself too deep at depth 4) and is left unchanged. The
+; whole-nest depth check fails, but because the fallback rescues a shallower
+; pair, no UnsupportedLoopNestDepth remark may accompany the Interchanged one.
+;-------------------------------------------------------------------------------
+define void @too_deep_reaches_shallow_pair() {
+entry:
+ br label %hk
+hk:
+ %k = phi i64 [0, %entry], [%kn, %lk]
+ br label %hi
+hi:
+ %i = phi i64 [0, %hk], [%in, %li]
+ br label %bx
+bx:
+ %x = phi i64 [0, %hi], [%xn, %bx]
+ %p = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @A, i64 0, i64 %k, i64 %x, i64 %i
+ %v = load double, ptr %p, align 8
+ %w = fadd double %v, 1.0
+ store double %w, ptr %p, align 8
+ %xn = add i64 %x, 1
+ %xe = icmp eq i64 %xn, 7
+ br i1 %xe, label %li, label %bx
+li:
+ %in = add i64 %i, 1
+ %ie = icmp eq i64 %in, 7
+ br i1 %ie, label %prea, label %hi
+prea:
+ br label %ha
+ha:
+ %a = phi i64 [0, %prea], [%an, %la]
+ br label %hb
+hb:
+ %b = phi i64 [0, %ha], [%bn, %lb]
+ br label %bc
+bc:
+ %c = phi i64 [0, %hb], [%cn, %bc]
+ %q = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @B, i64 0, i64 %a, i64 %c, i64 %b
+ %cv = load double, ptr %q, align 8
+ %cw = fadd double %cv, 1.0
+ store double %cw, ptr %q, align 8
+ %cn = add i64 %c, 1
+ %ce = icmp eq i64 %cn, 7
+ br i1 %ce, label %lb, label %bc
+lb:
+ %bn = add i64 %b, 1
+ %be = icmp eq i64 %bn, 7
+ br i1 %be, label %la, label %hb
+la:
+ %an = add i64 %a, 1
+ %ae = icmp eq i64 %an, 7
+ br i1 %ae, label %lk, label %ha
+lk:
+ %kn = add i64 %k, 1
+ %ke = icmp eq i64 %kn, 7
+ br i1 %ke, label %exit, label %hk
+exit:
+ ret void
+}
+
+; The hi/bx pair is swapped: the former inner header %bx now heads the loop
+; directly nested in %hk, and the former outer header %hi now heads the
+; innermost loop. The deeper ha/hb/bc triple, which has no eligible pair within
+; the cap, is untouched.
+; TOO-DEEP-LOOPS-LABEL: Loop info for function 'too_deep_reaches_shallow_pair':
+; TOO-DEEP-LOOPS: Loop at depth 1 containing: %hk<header>
+; TOO-DEEP-LOOPS-NEXT: Loop at depth 2 containing: %bx<header>
+; TOO-DEEP-LOOPS-NEXT: Loop at depth 3 containing: %hi<header>
+; TOO-DEEP-LOOPS-NEXT: Loop at depth 2 containing: %ha<header>
+; TOO-DEEP-LOOPS-NEXT: Loop at depth 3 containing: %hb<header>
+; TOO-DEEP-LOOPS-NEXT: Loop at depth 4 containing: %bc<header>
+;
+; Disabling fallback keeps both subtrees in their original order.
+; FALLBACK-DISABLED-LOOPS-LABEL: Loop info for function 'too_deep_reaches_shallow_pair':
+; FALLBACK-DISABLED-LOOPS: Loop at depth 1 containing: %hk<header>
+; FALLBACK-DISABLED-LOOPS-NEXT: Loop at depth 2 containing: %hi<header>
+; FALLBACK-DISABLED-LOOPS-NEXT: Loop at depth 3 containing: %bx<header>
+; FALLBACK-DISABLED-LOOPS-NEXT: Loop at depth 2 containing: %ha<header>
+; FALLBACK-DISABLED-LOOPS-NEXT: Loop at depth 3 containing: %hb<header>
+; FALLBACK-DISABLED-LOOPS-NEXT: Loop at depth 4 containing: %bc<header>
+
+;-------------------------------------------------------------------------------
+; Four eligible sibling-chain pairs under a common ancestor k (unused in any
+; index, so its surrounding dependence direction is unknown, as in
+; budget_exhausted_all_fail above), at four distinct depths: p3 (depth 3), p4
+; (depth 4, one wrapper level deep), p5 (depth 5, two wrapper levels deep), and
+; p6 (depth 6, three wrapper levels deep). With the budget at 2, retention
+; keeps only the Budget+1 deepest candidates while scanning (p6, p5, p4),
+; evicting the shallowest surplus (p3) outright the moment a 4th candidate (p6)
+; arrives and pushes the retained count 2 over budget; this exercises
+; Candidates.pop_back(). Of the retained three, only the two deepest (p6, p5)
+; are attempted, deepest first; p4 is kept solely to anchor the
+; budget-exhaustion remark.
+;-------------------------------------------------------------------------------
+define void @eviction_prunes_shallowest_surplus() !dbg !5 {
+entry:
+ br label %k.header
+
+k.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %k.latch ]
+ br label %p3.i.header
+
+p3.i.header:
+ %i3 = phi i64 [ 0, %k.header ], [ %i3.next, %p3.i.latch ]
+ br label %p3.j.body, !dbg !6
+
+p3.j.body:
+ %j3 = phi i64 [ 1, %p3.i.header ], [ %j3.next, %p3.j.body ]
+ %j3m1 = sub i64 %j3, 1
+ %p3.ld = getelementptr inbounds [8 x [8 x double]], ptr @DA, i64 0, i64 %j3, i64 %i3
+ %p3.v = load double, ptr %p3.ld, align 8
+ %p3.nv = fadd double %p3.v, 1.000000e+00
+ %p3.st = getelementptr inbounds [8 x [8 x double]], ptr @DA, i64 0, i64 %j3m1, i64 %i3
+ store double %p3.nv, ptr %p3.st, align 8
+ %j3.next = add i64 %j3, 1
+ %j3.ec = icmp eq i64 %j3.next, 8
+ br i1 %j3.ec, label %p3.i.latch, label %p3.j.body
+
+p3.i.latch:
+ %i3.next = add i64 %i3, 1
+ %i3.ec = icmp eq i64 %i3.next, 7
+ br i1 %i3.ec, label %m4.header, label %p3.i.header
+
+m4.header:
+ %m4 = phi i64 [ 0, %p3.i.latch ], [ %m4.next, %m4.latch ]
+ br label %p4.i.header
+
+p4.i.header:
+ %i4 = phi i64 [ 0, %m4.header ], [ %i4.next, %p4.i.latch ]
+ br label %p4.j.body, !dbg !7
+
+p4.j.body:
+ %j4 = phi i64 [ 1, %p4.i.header ], [ %j4.next, %p4.j.body ]
+ %j4m1 = sub i64 %j4, 1
+ %p4.ld = getelementptr inbounds [8 x [8 x double]], ptr @DB, i64 0, i64 %j4, i64 %i4
+ %p4.v = load double, ptr %p4.ld, align 8
+ %p4.nv = fadd double %p4.v, 1.000000e+00
+ %p4.st = getelementptr inbounds [8 x [8 x double]], ptr @DB, i64 0, i64 %j4m1, i64 %i4
+ store double %p4.nv, ptr %p4.st, align 8
+ %j4.next = add i64 %j4, 1
+ %j4.ec = icmp eq i64 %j4.next, 8
+ br i1 %j4.ec, label %p4.i.latch, label %p4.j.body
+
+p4.i.latch:
+ %i4.next = add i64 %i4, 1
+ %i4.ec = icmp eq i64 %i4.next, 7
+ br i1 %i4.ec, label %m4.latch, label %p4.i.header
+
+m4.latch:
+ %m4.next = add i64 %m4, 1
+ %m4.ec = icmp eq i64 %m4.next, 4
+ br i1 %m4.ec, label %m5a.header, label %m4.header
+
+m5a.header:
+ %m5a = phi i64 [ 0, %m4.latch ], [ %m5a.next, %m5a.latch ]
+ br label %m5b.header
+
+m5b.header:
+ %m5b = phi i64 [ 0, %m5a.header ], [ %m5b.next, %m5b.latch ]
+ br label %p5.i.header
+
+p5.i.header:
+ %i5 = phi i64 [ 0, %m5b.header ], [ %i5.next, %p5.i.latch ]
+ br label %p5.j.body, !dbg !8
+
+p5.j.body:
+ %j5 = phi i64 [ 1, %p5.i.header ], [ %j5.next, %p5.j.body ]
+ %j5m1 = sub i64 %j5, 1
+ %p5.ld = getelementptr inbounds [8 x [8 x double]], ptr @DC, i64 0, i64 %j5, i64 %i5
+ %p5.v = load double, ptr %p5.ld, align 8
+ %p5.nv = fadd double %p5.v, 1.000000e+00
+ %p5.st = getelementptr inbounds [8 x [8 x double]], ptr @DC, i64 0, i64 %j5m1, i64 %i5
+ store double %p5.nv, ptr %p5.st, align 8
+ %j5.next = add i64 %j5, 1
+ %j5.ec = icmp eq i64 %j5.next, 8
+ br i1 %j5.ec, label %p5.i.latch, label %p5.j.body
+
+p5.i.latch:
+ %i5.next = add i64 %i5, 1
+ %i5.ec = icmp eq i64 %i5.next, 7
+ br i1 %i5.ec, label %m5b.latch, label %p5.i.header
+
+m5b.latch:
+ %m5b.next = add i64 %m5b, 1
+ %m5b.ec = icmp eq i64 %m5b.next, 4
+ br i1 %m5b.ec, label %m5a.latch, label %m5b.header
+
+m5a.latch:
+ %m5a.next = add i64 %m5a, 1
+ %m5a.ec = icmp eq i64 %m5a.next, 4
+ br i1 %m5a.ec, label %m6a.header, label %m5a.header
+
+m6a.header:
+ %m6a = phi i64 [ 0, %m5a.latch ], [ %m6a.next, %m6a.latch ]
+ br label %m6b.header
+
+m6b.header:
+ %m6b = phi i64 [ 0, %m6a.header ], [ %m6b.next, %m6b.latch ]
+ br label %m6c.header
+
+m6c.header:
+ %m6c = phi i64 [ 0, %m6b.header ], [ %m6c.next, %m6c.latch ]
+ br label %p6.i.header
+
+p6.i.header:
+ %i6 = phi i64 [ 0, %m6c.header ], [ %i6.next, %p6.i.latch ]
+ br label %p6.j.body, !dbg !9
+
+p6.j.body:
+ %j6 = phi i64 [ 1, %p6.i.header ], [ %j6.next, %p6.j.body ]
+ %j6m1 = sub i64 %j6, 1
+ %p6.ld = getelementptr inbounds [8 x [8 x double]], ptr @DD, i64 0, i64 %j6, i64 %i6
+ %p6.v = load double, ptr %p6.ld, align 8
+ %p6.nv = fadd double %p6.v, 1.000000e+00
+ %p6.st = getelementptr inbounds [8 x [8 x double]], ptr @DD, i64 0, i64 %j6m1, i64 %i6
+ store double %p6.nv, ptr %p6.st, align 8
+ %j6.next = add i64 %j6, 1
+ %j6.ec = icmp eq i64 %j6.next, 8
+ br i1 %j6.ec, label %p6.i.latch, label %p6.j.body
+
+p6.i.latch:
+ %i6.next = add i64 %i6, 1
+ %i6.ec = icmp eq i64 %i6.next, 7
+ br i1 %i6.ec, label %m6c.latch, label %p6.i.header
+
+m6c.latch:
+ %m6c.next = add i64 %m6c, 1
+ %m6c.ec = icmp eq i64 %m6c.next, 4
+ br i1 %m6c.ec, label %m6b.latch, label %m6c.header
+
+m6b.latch:
+ %m6b.next = add i64 %m6b, 1
+ %m6b.ec = icmp eq i64 %m6b.next, 4
+ br i1 %m6b.ec, label %m6a.latch, label %m6b.header
+
+m6a.latch:
+ %m6a.next = add i64 %m6a, 1
+ %m6a.ec = icmp eq i64 %m6a.next, 4
+ br i1 %m6a.ec, label %k.latch, label %m6a.header
+
+k.latch:
+ %k.next = add i64 %k, 1
+ %k.ec = icmp eq i64 %k.next, 4
+ br i1 %k.ec, label %exit, label %k.header
+
+exit:
+ ret void
+}
+
+; Eviction leaves every candidate pair unswapped and in its original sibling
+; order: the evicted depth-3 pair (p3), the unattempted-anchor depth-4 pair
+; (p4), and the two attempted, still-rejected depth-5 and depth-6 pairs (p5,
+; p6) all keep their original outer-then-inner header order.
+; EVICT-LOOPS-LABEL: Loop info for function 'eviction_prunes_shallowest_surplus':
+; EVICT-LOOPS: Loop at depth 1 containing: %k.header<header>
+; EVICT-LOOPS-NEXT: Loop at depth 2 containing: %p3.i.header<header>
+; EVICT-LOOPS-NEXT: Loop at depth 3 containing: %p3.j.body<header>
+; EVICT-LOOPS-NEXT: Loop at depth 2 containing: %m4.header<header>
+; EVICT-LOOPS-NEXT: Loop at depth 3 containing: %p4.i.header<header>
+; EVICT-LOOPS-NEXT: Loop at depth 4 containing: %p4.j.body<header>
+; EVICT-LOOPS-NEXT: Loop at depth 2 containing: %m5a.header<header>
+; EVICT-LOOPS-NEXT: Loop at depth 3 containing: %m5b.header<header>
+; EVICT-LOOPS-NEXT: Loop at depth 4 containing: %p5.i.header<header>
+; EVICT-LOOPS-NEXT: Loop at depth 5 containing: %p5.j.body<header>
+; EVICT-LOOPS-NEXT: Loop at depth 2 containing: %m6a.header<header>
+; EVICT-LOOPS-NEXT: Loop at depth 3 containing: %m6b.header<header>
+; EVICT-LOOPS-NEXT: Loop at depth 4 containing: %m6c.header<header>
+; EVICT-LOOPS-NEXT: Loop at depth 5 containing: %p6.i.header<header>
+; EVICT-LOOPS-NEXT: Loop at depth 6 containing: %p6.j.body<header>
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "llvm", isOptimized: true, runtimeVersion: 0, emissionKind: LineTablesOnly)
+!1 = !DIFile(filename: "inner-subnest-budget.ll", directory: "")
+!2 = !{}
+!3 = !{i32 2, !"Debug Info Version", i32 3}
+!4 = !DISubroutineType(types: !2)
+!5 = distinct !DISubprogram(name: "eviction_prunes_shallowest_surplus", scope: !1, file: !1, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !2)
+!6 = !DILocation(line: 30, column: 1, scope: !5)
+!7 = !DILocation(line: 40, column: 1, scope: !5)
+!8 = !DILocation(line: 50, column: 1, scope: !5)
+!9 = !DILocation(line: 60, column: 1, scope: !5)
+!10 = distinct !DISubprogram(name: "budget_exhausted_all_fail", scope: !1, file: !1, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !2)
+!11 = !DILocation(line: 70, column: 1, scope: !10)
+!12 = !DILocation(line: 80, column: 1, scope: !10)
+!13 = !DILocation(line: 90, column: 1, scope: !10)
diff --git a/llvm/test/Transforms/LoopInterchange/inner-subnest-cache-cost.ll b/llvm/test/Transforms/LoopInterchange/inner-subnest-cache-cost.ll
new file mode 100644
index 0000000000000..ac8b62ee71ce9
--- /dev/null
+++ b/llvm/test/Transforms/LoopInterchange/inner-subnest-cache-cost.ll
@@ -0,0 +1,372 @@
+; Cache profitability for a fallback pair must use that pair's root-to-leaf
+; ancestor chain and exclude references from disjoint sibling subtrees.
+;
+; RUN: llvm-extract -S -func=disjoint_costmodel %s -o %t.disjoint
+; RUN: opt < %t.disjoint -passes=loop-interchange -cache-line-size=64 \
+; RUN: -loop-interchange-profitabilities=cache \
+; RUN: -verify-dom-info -verify-loop-info -verify-scev -verify-loop-lcssa \
+; RUN: -pass-remarks-output=%t.disjoint.yaml -disable-output
+; RUN: FileCheck %s --check-prefix=DISJOINT \
+; RUN: --input-file=%t.disjoint.yaml --implicit-check-not=Interchanged
+;
+; RUN: llvm-extract -S -func=control_same_pair_alone %s -o %t.control-negative
+; RUN: opt < %t.control-negative -passes=loop-interchange -cache-line-size=64 \
+; RUN: -loop-interchange-profitabilities=cache \
+; RUN: -verify-dom-info -verify-loop-info -verify-scev -verify-loop-lcssa \
+; RUN: -pass-remarks-output=%t.control-negative.yaml -disable-output
+; RUN: FileCheck %s --check-prefix=CONTROL-NEGATIVE \
+; RUN: --input-file=%t.control-negative.yaml \
+; RUN: --implicit-check-not=Interchanged
+;
+; RUN: llvm-extract -S -func=fallback_cache_profitable %s -o %t.fallback-positive
+; RUN: opt < %t.fallback-positive -passes=loop-interchange -cache-line-size=64 \
+; RUN: -loop-interchange-profitabilities=cache \
+; RUN: -verify-dom-info -verify-loop-info -verify-scev -verify-loop-lcssa \
+; RUN: -pass-remarks-output=%t.fallback-positive.yaml -disable-output
+; RUN: FileCheck %s --check-prefix=FALLBACK-POSITIVE \
+; RUN: --input-file=%t.fallback-positive.yaml
+;
+; RUN: llvm-extract -S -func=linear_cache_profitable %s -o %t.linear-positive
+; RUN: opt < %t.linear-positive -passes=loop-interchange -cache-line-size=64 \
+; RUN: -loop-interchange-profitabilities=cache \
+; RUN: -verify-dom-info -verify-loop-info -verify-scev -verify-loop-lcssa \
+; RUN: -pass-remarks-output=%t.linear-positive.yaml -disable-output
+; RUN: FileCheck %s --check-prefix=LINEAR-POSITIVE \
+; RUN: --input-file=%t.linear-positive.yaml
+;
+; RUN: llvm-extract -S -func=two_equal_depth_leaves %s -o %t.equal-depth
+; RUN: opt < %t.equal-depth -passes=loop-interchange -cache-line-size=64 \
+; RUN: -loop-interchange-profitabilities=cache \
+; RUN: -verify-dom-info -verify-loop-info -verify-scev -verify-loop-lcssa \
+; RUN: -pass-remarks-output=%t.equal-depth.yaml -disable-output
+; RUN: FileCheck %s --check-prefix=EQUAL-DEPTH \
+; RUN: --input-file=%t.equal-depth.yaml
+;
+; RUN: opt < %s -passes='loop(loop-interchange),print<loops>' \
+; RUN: -cache-line-size=64 -loop-interchange-profitabilities=cache \
+; RUN: -disable-output 2>&1 | FileCheck %s --check-prefix=LOOPS
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+
+ at X = global [4 x [1335 x [100 x double]]] zeroinitializer
+ at Y = global [4 x [4 x [1335 x [1335 x double]]]] zeroinitializer
+ at P = global [4 x [128 x [128 x double]]] zeroinitializer
+ at S = global [4 x [128 x double]] zeroinitializer
+
+; The d/e pair is rejected for dependence. The fallback then evaluates a/b,
+; whose inner j loop is already unit stride. The disjoint c/d/e subtree must
+; not make a/b appear profitable.
+;
+; DISJOINT: --- !Missed
+; DISJOINT: Name: Dependence
+; DISJOINT-NEXT: Function: disjoint_costmodel
+; DISJOINT: --- !Missed
+; DISJOINT: Name: InterchangeNotProfitable
+; DISJOINT-NEXT: Function: disjoint_costmodel
+; DISJOINT: - String: Interchanging loops is not considered to improve cache locality nor vectorization.
+define void @disjoint_costmodel() {
+entry:
+ br label %r.header
+
+r.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %r.latch ]
+ br label %a.header
+
+a.header:
+ %i = phi i64 [ 0, %r.header ], [ %i.next, %a.latch ]
+ br label %b.body
+
+b.body:
+ %j = phi i64 [ 0, %a.header ], [ %j.next, %b.body ]
+ %xp = getelementptr inbounds [4 x [1335 x [100 x double]]],
+ ptr @X, i64 0, i64 %k, i64 %i, i64 %j
+ %xv = load double, ptr %xp, align 8
+ %xa = fadd double %xv, 1.000000e+00
+ store double %xa, ptr %xp, align 8
+ %j.next = add i64 %j, 1
+ %j.done = icmp eq i64 %j.next, 100
+ br i1 %j.done, label %a.latch, label %b.body
+
+a.latch:
+ %i.next = add i64 %i, 1
+ %i.done = icmp eq i64 %i.next, 1335
+ br i1 %i.done, label %c.header, label %a.header
+
+c.header:
+ %p = phi i64 [ 0, %a.latch ], [ %p.next, %c.latch ]
+ br label %d.header
+
+d.header:
+ %q = phi i64 [ 1, %c.header ], [ %q.next, %d.latch ]
+ br label %e.body
+
+e.body:
+ %w = phi i64 [ 0, %d.header ], [ %w.next, %e.body ]
+ %q.prev = add i64 %q, -1
+ %w.next.index = add i64 %w, 1
+ %ysrc = getelementptr inbounds [4 x [4 x [1335 x [1335 x double]]]],
+ ptr @Y, i64 0, i64 %k, i64 %p, i64 %q.prev, i64 %w.next.index
+ %yv = load double, ptr %ysrc, align 8
+ %ya = fadd double %yv, 1.000000e+00
+ %ydst = getelementptr inbounds [4 x [4 x [1335 x [1335 x double]]]],
+ ptr @Y, i64 0, i64 %k, i64 %p, i64 %q, i64 %w
+ store double %ya, ptr %ydst, align 8
+ %w.next = add i64 %w, 1
+ %w.done = icmp eq i64 %w.next, 1334
+ br i1 %w.done, label %d.latch, label %e.body
+
+d.latch:
+ %q.next = add i64 %q, 1
+ %q.done = icmp eq i64 %q.next, 1335
+ br i1 %q.done, label %c.latch, label %d.header
+
+c.latch:
+ %p.next = add i64 %p, 1
+ %p.done = icmp eq i64 %p.next, 4
+ br i1 %p.done, label %r.latch, label %c.header
+
+r.latch:
+ %k.next = add i64 %k, 1
+ %k.done = icmp eq i64 %k.next, 4
+ br i1 %k.done, label %exit, label %r.header
+
+exit:
+ ret void
+}
+
+; The identical pair on the standard path has a complete cache analysis and
+; declines because interchange would not improve locality.
+;
+; CONTROL-NEGATIVE: --- !Missed
+; CONTROL-NEGATIVE: Name: InterchangeNotProfitable
+; CONTROL-NEGATIVE-NEXT: Function: control_same_pair_alone
+; CONTROL-NEGATIVE: - String: Interchanging loops is not considered to improve cache locality nor vectorization.
+define void @control_same_pair_alone() {
+entry:
+ br label %r.header
+
+r.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %r.latch ]
+ br label %a.header
+
+a.header:
+ %i = phi i64 [ 0, %r.header ], [ %i.next, %a.latch ]
+ br label %b.body
+
+b.body:
+ %j = phi i64 [ 0, %a.header ], [ %j.next, %b.body ]
+ %xp = getelementptr inbounds [4 x [1335 x [100 x double]]],
+ ptr @X, i64 0, i64 %k, i64 %i, i64 %j
+ %xv = load double, ptr %xp, align 8
+ %xa = fadd double %xv, 1.000000e+00
+ store double %xa, ptr %xp, align 8
+ %j.next = add i64 %j, 1
+ %j.done = icmp eq i64 %j.next, 100
+ br i1 %j.done, label %a.latch, label %b.body
+
+a.latch:
+ %i.next = add i64 %i, 1
+ %i.done = icmp eq i64 %i.next, 1335
+ br i1 %i.done, label %r.latch, label %a.header
+
+r.latch:
+ %k.next = add i64 %k, 1
+ %k.done = icmp eq i64 %k.next, 4
+ br i1 %k.done, label %exit, label %r.header
+
+exit:
+ ret void
+}
+
+; The candidate leaf supplies the candidate chain's cache reference groups, so
+; the fallback keeps the same positive cache verdict as the linear control.
+;
+; FALLBACK-POSITIVE: --- !Passed
+; FALLBACK-POSITIVE: Name: Interchanged
+; FALLBACK-POSITIVE-NEXT: Function: fallback_cache_profitable
+define void @fallback_cache_profitable() {
+entry:
+ br label %root.header
+
+root.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %root.latch ]
+ br label %outer.header
+
+outer.header:
+ %i = phi i64 [ 0, %root.header ], [ %i.next, %outer.latch ]
+ br label %inner.body
+
+inner.body:
+ %j = phi i64 [ 0, %outer.header ], [ %j.next, %inner.body ]
+ %p = getelementptr inbounds [4 x [128 x [128 x double]]],
+ ptr @P, i64 0, i64 %k, i64 %j, i64 %i
+ %v = load double, ptr %p, align 8
+ %next = fadd double %v, 1.000000e+00
+ store double %next, ptr %p, align 8
+ %j.next = add i64 %j, 1
+ %j.done = icmp eq i64 %j.next, 128
+ br i1 %j.done, label %outer.latch, label %inner.body
+
+outer.latch:
+ %i.next = add i64 %i, 1
+ %i.done = icmp eq i64 %i.next, 128
+ br i1 %i.done, label %sibling.preheader, label %outer.header
+
+sibling.preheader:
+ br label %sibling.body
+
+sibling.body:
+ %s = phi i64 [ 0, %sibling.preheader ], [ %s.next, %sibling.body ]
+ %sp = getelementptr inbounds [4 x [128 x double]],
+ ptr @S, i64 0, i64 %k, i64 %s
+ store double 1.000000e+00, ptr %sp, align 8
+ %s.next = add i64 %s, 1
+ %s.done = icmp eq i64 %s.next, 128
+ br i1 %s.done, label %root.latch, label %sibling.body
+
+root.latch:
+ %k.next = add i64 %k, 1
+ %k.done = icmp eq i64 %k.next, 4
+ br i1 %k.done, label %exit, label %root.header
+
+exit:
+ ret void
+}
+
+; LINEAR-POSITIVE: --- !Passed
+; LINEAR-POSITIVE: Name: Interchanged
+; LINEAR-POSITIVE-NEXT: Function: linear_cache_profitable
+define void @linear_cache_profitable() {
+entry:
+ br label %root.header
+
+root.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %root.latch ]
+ br label %outer.header
+
+outer.header:
+ %i = phi i64 [ 0, %root.header ], [ %i.next, %outer.latch ]
+ br label %inner.body
+
+inner.body:
+ %j = phi i64 [ 0, %outer.header ], [ %j.next, %inner.body ]
+ %p = getelementptr inbounds [4 x [128 x [128 x double]]],
+ ptr @P, i64 0, i64 %k, i64 %j, i64 %i
+ %v = load double, ptr %p, align 8
+ %next = fadd double %v, 1.000000e+00
+ store double %next, ptr %p, align 8
+ %j.next = add i64 %j, 1
+ %j.done = icmp eq i64 %j.next, 128
+ br i1 %j.done, label %outer.latch, label %inner.body
+
+outer.latch:
+ %i.next = add i64 %i, 1
+ %i.done = icmp eq i64 %i.next, 128
+ br i1 %i.done, label %root.latch, label %outer.header
+
+root.latch:
+ %k.next = add i64 %k, 1
+ %k.done = icmp eq i64 %k.next, 4
+ br i1 %k.done, label %exit, label %root.header
+
+exit:
+ ret void
+}
+
+; Both candidate leaves have the same depth. The first pair is not profitable;
+; the second pair is profitable from its own reference groups and interchanges.
+;
+; EQUAL-DEPTH: --- !Missed
+; EQUAL-DEPTH: Name: InterchangeNotProfitable
+; EQUAL-DEPTH-NEXT: Function: two_equal_depth_leaves
+; EQUAL-DEPTH: --- !Passed
+; EQUAL-DEPTH: Name: Interchanged
+; EQUAL-DEPTH-NEXT: Function: two_equal_depth_leaves
+define void @two_equal_depth_leaves() {
+entry:
+ br label %root.header
+
+root.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %root.latch ]
+ br label %first.outer
+
+first.outer:
+ %i = phi i64 [ 0, %root.header ], [ %i.next, %first.outer.latch ]
+ br label %first.inner
+
+first.inner:
+ %j = phi i64 [ 0, %first.outer ], [ %j.next, %first.inner ]
+ %xp = getelementptr inbounds [4 x [1335 x [100 x double]]],
+ ptr @X, i64 0, i64 %k, i64 %i, i64 %j
+ %xv = load double, ptr %xp, align 8
+ %xa = fadd double %xv, 1.000000e+00
+ store double %xa, ptr %xp, align 8
+ %j.next = add i64 %j, 1
+ %j.done = icmp eq i64 %j.next, 100
+ br i1 %j.done, label %first.outer.latch, label %first.inner
+
+first.outer.latch:
+ %i.next = add i64 %i, 1
+ %i.done = icmp eq i64 %i.next, 1335
+ br i1 %i.done, label %second.preheader, label %first.outer
+
+second.preheader:
+ br label %second.outer
+
+second.outer:
+ %i2 = phi i64 [ 0, %second.preheader ], [ %i2.next, %second.outer.latch ]
+ br label %second.inner
+
+second.inner:
+ %j2 = phi i64 [ 0, %second.outer ], [ %j2.next, %second.inner ]
+ %pp = getelementptr inbounds [4 x [128 x [128 x double]]],
+ ptr @P, i64 0, i64 %k, i64 %j2, i64 %i2
+ %pv = load double, ptr %pp, align 8
+ %pa = fadd double %pv, 1.000000e+00
+ store double %pa, ptr %pp, align 8
+ %j2.next = add i64 %j2, 1
+ %j2.done = icmp eq i64 %j2.next, 128
+ br i1 %j2.done, label %second.outer.latch, label %second.inner
+
+second.outer.latch:
+ %i2.next = add i64 %i2, 1
+ %i2.done = icmp eq i64 %i2.next, 128
+ br i1 %i2.done, label %root.latch, label %second.outer
+
+root.latch:
+ %k.next = add i64 %k, 1
+ %k.done = icmp eq i64 %k.next, 4
+ br i1 %k.done, label %exit, label %root.header
+
+exit:
+ ret void
+}
+
+; Cache-negative fallback candidates keep their original order; profitable
+; fallback candidates interchange.
+; LOOPS-LABEL: Loop info for function 'disjoint_costmodel':
+; LOOPS: Loop at depth 1 containing: %r.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %a.header<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %b.body<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %c.header<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %d.header<header>
+; LOOPS-NEXT: Loop at depth 4 containing: %e.body<header>
+; LOOPS-LABEL: Loop info for function 'control_same_pair_alone':
+; LOOPS: Loop at depth 1 containing: %r.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %a.header<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %b.body<header>
+; LOOPS-LABEL: Loop info for function 'fallback_cache_profitable':
+; LOOPS: Loop at depth 1 containing: %root.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %inner.body<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %outer.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %sibling.body<header>
+; LOOPS-LABEL: Loop info for function 'linear_cache_profitable':
+; LOOPS: Loop at depth 1 containing: %root.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %inner.body<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %outer.header<header>
+; LOOPS-LABEL: Loop info for function 'two_equal_depth_leaves':
+; LOOPS: Loop at depth 1 containing: %root.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %first.outer<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %first.inner<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %second.inner<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %second.outer<header>
diff --git a/llvm/test/Transforms/LoopInterchange/inner-subnest-candidates.ll b/llvm/test/Transforms/LoopInterchange/inner-subnest-candidates.ll
index 0fc41f3f0645f..1233e64cc1b64 100644
--- a/llvm/test/Transforms/LoopInterchange/inner-subnest-candidates.ll
+++ b/llvm/test/Transforms/LoopInterchange/inner-subnest-candidates.ll
@@ -1,64 +1,100 @@
-; Precommit test for inner-subnest candidate formation in LoopInterchange.
+; Behavior test for inner-subnest candidate formation in LoopInterchange.
;
-; This file is entirely hand-maintained. Do NOT run update_test_checks.py on
-; it: the CHECK lines deliberately pin the *current* no-transform behavior of
-; the unmodified LoopInterchange pass so that the follow-on
-; candidate-formation implementation can show a reviewable diff.
+; This file is entirely hand-maintained. Do NOT run update_test_checks.py on it.
;
; Background:
; LoopInterchangePass::run consumes LoopNest::getLoops(), which is a
-; *breadth-first* walk over every descendant loop (siblings included). The
-; pass then (1) rejects when that flat list is longer than
-; MaxLoopNestDepth (=10), (2) rejects when any member has an uncomputable
-; backedge / non-unique exit, and (3) in LoopInterchange::run(LoopNest&)
-; rejects when the flat list is not a single linear chain. In each of those
-; cases the pass bails *before* it ever reaches an otherwise eligible,
-; profitable adjacent parent/child pair, so no interchange happens today.
+; *breadth-first* walk over every descendant loop (siblings included).
+; The pass (1) uses LoopNest::getNestDepth() for the depth policy instead of
+; that descendant count, so a shallow sibling-rich nest is not misclassified
+; as too deep; and (2) when the whole breadth-first nest is unsuitable for
+; the standard multi-swap path -- it is non-linear, a loop in its linear
+; chain is uncomputable or unsupported, or it exceeds the depth policy --
+; enumerates direct parent/leaf-child edges (a parent with a single child
+; loop that is itself a leaf) and performs at most one sound, profitable
+; interchange through the inner-subnest fallback. The standard multi-swap
+; path and its behavior are unchanged.
;
-; A fixed leading dimension of 1335 doubles gives a 10,680-byte inner stride
-; (1335 * 8), a cache-hostile column-major access typical of shallow-water
-; stencil benchmarks. The two `admitted_*` functions below are standalone,
-; admissible 2-deep nests that the *current* pass already interchanges under
-; default profitability; they establish that the shared candidate pair is legal
-; and profitable "once admitted". Every other function embeds that same shape
-; (or a deliberately-permanent negative) inside an enclosing structure that the
-; current pass rejects, and pins that the pair is left in its original order.
+; A fixed leading dimension of 1335 doubles reproduces canonical SWIM's
+; 10,680-byte inner stride (1335 * 8). The two `admitted_*` functions below are
+; standalone, admissible 2-deep nests that the *standard* path interchanges
+; under default profitability; they establish that the shared candidate pair is
+; legal and profitable "once admitted". Five positive fixtures embed that
+; same shape inside an enclosing structure the standard path rejects; the
+; fallback reaches and interchanges the eligible pair there, where the
+; unmodified pass left the nest alone. The lasting negatives stay in their
+; original order.
;
-; This precommit only asserts behavior observable on the unmodified pass: the
-; breadth-first depth remark, the applied/analysis/missed remarks, and the
-; actual unswapped IR. Direct-edge candidate selection, ancestor-column
-; dependence handling and sibling exclusion are oracles for the follow-on
-; candidate-formation implementation and are intentionally NOT asserted here.
+; The applied/analysis/missed remark stream (YAML) is the primary decision and
+; diagnostic oracle. The IR run additionally pins that surviving structure
+; (address expressions, reassociated reductions, sibling traffic) is preserved
+; and that the analysis verifiers pass. Because the "Interchanged" remark is emitted
+; *before* the transform runs and carries no loop identity, a third run pins the
+; produced loop hierarchy with `print<loops>`: after a fallback interchange the
+; former inner header block heads the new outer loop and the former outer header
+; block heads the new inner loop, which distinguishes a real swap from an
+; unswapped nest and, for sibling-rich parents, proves the parent's sibling
+; program order is preserved (see the LOOPS lines).
;
; RUN: opt < %s -passes=loop-interchange -cache-line-size=64 \
; RUN: -verify-dom-info -verify-loop-info -verify-scev -verify-loop-lcssa \
; RUN: -S 2>&1 | FileCheck %s --check-prefix=IR
;
-; Full, function-associated remark log (Passed / Missed / Analysis).
+; Post-transform loop hierarchy. LoopInterchange preserves LoopAnalysis, so the
+; following print<loops> reflects the pass's incrementally-updated loop tree
+; (headers, nesting, and sibling order), not a fresh rebuild.
+; RUN: opt < %s -passes='loop(loop-interchange),print<loops>' -cache-line-size=64 \
+; RUN: -disable-output 2>&1 | FileCheck %s --check-prefix=LOOPS
+;
+; Full, function-associated remark log (Passed / Missed / Analysis). The
+; --implicit-check-not proves the depth misclassification is fixed: with the
+; true nesting depth used for the policy, no function is rejected for depth.
; RUN: opt < %s -passes=loop-interchange -cache-line-size=64 \
; RUN: -pass-remarks=loop-interchange -pass-remarks-missed=loop-interchange \
; RUN: -pass-remarks-output=%t -disable-output
-; RUN: FileCheck %s --check-prefix=YAML --input-file=%t
-;
-; Raising the depth cap past the breadth-first count removes the (false) depth
-; rejection for the shallow, sibling-rich nest, proving the count -- not a real
-; depth-3 problem -- triggered it. Under the raised cap bfs_loop_count_is_not_depth
-; clears the depth gate and reaches dependence analysis (its own function-
-; associated !Analysis Dependence record) instead of UnsupportedLoopNestDepth; it
-; still does not interchange (its next blocker is non-linearity), which the
-; follow-on candidate-formation implementation addresses. Proven from the
-; function-associated YAML remark stream, not a shared stderr string that
-; another function could satisfy.
-; RUN: opt < %s -passes=loop-interchange -cache-line-size=64 \
-; RUN: -loop-interchange-max-loop-nest-depth=32 \
-; RUN: -pass-remarks-output=%t.raised -disable-output
-; RUN: FileCheck %s --check-prefix=RAISED --input-file=%t.raised \
+; RUN: FileCheck %s --check-prefix=YAML --input-file=%t \
; RUN: --implicit-check-not=UnsupportedLoopNestDepth
+;
+; The switch gates fallback processing, not the true-depth policy. With
+; fallback disabled, this non-linear nest is no longer rejected for depth and
+; is not interchanged, so it emits an empty remark file. `test -e` makes the
+; emptiness check non-vacuous.
+; RUN: llvm-extract -S -func=bfs_loop_count_is_not_depth %s \
+; RUN: -o %t.disabled-nonlinear
+; RUN: opt < %t.disabled-nonlinear -passes=loop-interchange \
+; RUN: -cache-line-size=64 \
+; RUN: -loop-interchange-enable-inner-subnest-fallback=false \
+; RUN: -pass-remarks-output=%t.disabled-nonlinear.yaml -disable-output
+; RUN: test -e %t.disabled-nonlinear.yaml
+; RUN: test ! -s %t.disabled-nonlinear.yaml
+; RUN: opt < %t.disabled-nonlinear \
+; RUN: -passes='loop(loop-interchange),print<loops>' -cache-line-size=64 \
+; RUN: -loop-interchange-enable-inner-subnest-fallback=false \
+; RUN: -disable-output 2>&1 | \
+; RUN: FileCheck %s --check-prefix=ORIGINAL-NONLINEAR-LOOPS
+;
+; RUN: llvm-extract -S -func=uncomputable_sibling_does_not_block %s \
+; RUN: -o %t.disabled-uncomputable
+; RUN: opt < %t.disabled-uncomputable -passes=loop-interchange \
+; RUN: -cache-line-size=64 \
+; RUN: -loop-interchange-enable-inner-subnest-fallback=false \
+; RUN: -pass-remarks-output=%t.disabled-uncomputable.yaml -disable-output
+; RUN: test -e %t.disabled-uncomputable.yaml
+; RUN: test ! -s %t.disabled-uncomputable.yaml
+; RUN: opt < %t.disabled-uncomputable \
+; RUN: -passes='loop(loop-interchange),print<loops>' -cache-line-size=64 \
+; RUN: -loop-interchange-enable-inner-subnest-fallback=false \
+; RUN: -disable-output 2>&1 | \
+; RUN: FileCheck %s --check-prefix=DISABLED-UNCOMPUTABLE-LOOPS
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
;-------------------------------------------------------------------------------
-; Expected current remark log (function/module order). See per-function notes.
+; Expected remark log (function/module order). See per-function notes.
+; The two admitted_* controls interchange through the standard path; five
+; positive fixtures interchange through the inner-subnest fallback. Lasting
+; negatives are rejected by profitability, strict reductions, an unsupported
+; pair, or failure to form an eligible leaf candidate.
;-------------------------------------------------------------------------------
; YAML: --- !Analysis
; YAML: Name: Dependence
@@ -72,51 +108,71 @@ target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
; YAML: --- !Passed
; YAML: Name: Interchanged
; YAML: Function: admitted_1335_pair_three_reductions
-; YAML: --- !Missed
-; YAML: Name: UnsupportedLoopNestDepth
+; bfs_loop_count_is_not_depth is no longer rejected for depth (true depth 3)
+; and is interchanged through the fallback.
+; YAML: --- !Passed
+; YAML: Name: Interchanged
; YAML: Function: bfs_loop_count_is_not_depth
-; The two uncomputable-neighbour nests bail before the analysis remark, so they
-; emit nothing at all between the depth miss and the two-candidate analysis.
-; YAML-NOT: Function: uncomputable_sibling_does_not_block
-; YAML-NOT: Function: uncomputable_ancestor_partition
-; YAML: --- !Analysis
-; YAML: Name: Dependence
+; The non-linear uncomputable sibling reaches fallback directly. The linear
+; uncomputable ancestor exercises the whole-nest computability rejection. Both
+; interchange their computable inner pair without an analysis remark.
+; YAML: --- !Passed
+; YAML: Name: Interchanged
+; YAML: Function: uncomputable_sibling_does_not_block
+; YAML: --- !Passed
+; YAML: Name: Interchanged
+; YAML: Function: uncomputable_ancestor_partition
+; two_candidate_pairs_one_fallback has two eligible pairs; exactly one (the
+; deterministic first) is interchanged through the fallback.
+; YAML: --- !Passed
+; YAML: Name: Interchanged
; YAML: Function: two_candidate_pairs_one_fallback
+; YAML-NOT: Function: two_candidate_pairs_one_fallback
+; partly_exact_reduction is a plain admissible 2-deep nest on the standard path;
+; its strict fadd is rejected exactly as before.
; YAML: --- !Analysis
; YAML: Name: Dependence
; YAML: Function: partly_exact_reduction
; YAML: --- !Missed
; YAML: Name: UnsupportedPHIOuter
; YAML: Function: partly_exact_reduction
-; YAML: --- !Analysis
-; YAML: Name: Dependence
+; The dynamic leading dimension is reached by the fallback but declined by
+; default profitability.
+; YAML: --- !Missed
+; YAML: Name: InterchangeNotProfitable
; YAML: Function: dynamic_leading_dimension_subnest
-; YAML: --- !Analysis
-; YAML: Name: Dependence
+; The all-exact and partly-exact subnests are reached by the fallback but their
+; strict recurrences are rejected.
+; YAML: --- !Missed
+; YAML: Name: UnsupportedPHIOuter
; YAML: Function: all_exact_reduction_subnest
-; YAML: --- !Analysis
-; YAML: Name: Dependence
+; YAML: --- !Missed
+; YAML: Name: UnsupportedPHIOuter
; YAML: Function: partly_exact_reduction_subnest
-; YAML: --- !Analysis
-; YAML: Name: Dependence
-; YAML: Function: non_leaf_candidate_subnest
-
-; The two admitted controls still interchange (two Interchanged records); the very
-; next analysis record is bfs_loop_count_is_not_depth, proving it now clears the
-; raised depth gate and reaches dependence analysis rather than being rejected as
-; too deep. --implicit-check-not proves no UnsupportedLoopNestDepth is emitted.
-; RAISED: Name: Interchanged
-; RAISED: Name: Interchanged
-; RAISED: Name: Dependence
-; RAISED-NEXT: Function: bfs_loop_count_is_not_depth
+; non_leaf_candidate_subnest forms no eligible candidate (its only single-child
+; edge has a non-leaf inner loop), so the fallback leaves it unchanged.
+; distinct_depth_deepest_first has two eligible pairs at different depths; the
+; deeper pair (dB, inner depth 4) is selected deepest-first and interchanged, and
+; the shallower pair (sA) is left untouched (at most one transform).
+; YAML: --- !Passed
+; YAML: Name: Interchanged
+; YAML: Function: distinct_depth_deepest_first
+; YAML-NOT: Function: distinct_depth_deepest_first
+; unsupported_pair_backedge_subnest's non-linear list reaches fallback
+; directly. Its i/j pair has two exiting blocks (no unique exit), so pair-local
+; computability rejects it as unsupported without an analysis remark.
+; YAML: --- !Missed
+; YAML: Name: FallbackUnsupportedPair
+; YAML: Function: unsupported_pair_backedge_subnest
+;
;-------------------------------------------------------------------------------
; Positive controls: the shared fixed-1335 reduction pair, presented as a plain
-; admissible 2-deep nest, is interchanged by the current pass under default
+; admissible 2-deep nest, is interchanged by the standard path under default
; profitability. These prove the pair is legal + profitable "once admitted", so
-; every blocked case below fails only because of its enclosing structure.
-; Their transformed IR is not pinned here (that belongs to the follow-on
-; behavior commit's before/after); the Passed remark above is the oracle.
+; every fallback case below is reached and interchanged (or rejected) purely on
+; its own merits. The Passed records are selection/invocation oracles; the LOOPS
+; checks pin the resulting hierarchy.
;-------------------------------------------------------------------------------
; double sum = 0; for i: for j: sum += A[j][i]; (inner j strides 1335 doubles)
@@ -151,8 +207,17 @@ exit:
ret void
}
-; Three independent reassociated reductions over three arrays, the shape of a
-; multi-array checksum loop.
+; Standard-path control: this admissible 2-deep nest is interchanged by the
+; multi-swap path, not the fallback. After the swap the former inner header
+; %inner.header heads the new outer loop and the former outer header
+; %outer.header heads the new inner loop (a real swap, not just a surviving
+; nest). This also guards that the standard multi-swap path is unchanged by the
+; inner-subnest fallback.
+; LOOPS-LABEL: Loop info for function 'admitted_1335_pair_one_reduction':
+; LOOPS: Loop at depth 1 containing: %inner.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %outer.header<header>
+
+; Three independent reassociated reductions, matching the SWIM checksum shape.
define void @admitted_1335_pair_three_reductions(ptr %A, ptr %B, ptr %C, ptr %R) {
entry:
br label %outer.header
@@ -202,11 +267,20 @@ exit:
ret void
}
+; Standard-path control (three reductions). Same swap oracle as the one-reduction
+; control: former inner header %inner.header heads the new outer loop.
+; LOOPS-LABEL: Loop info for function 'admitted_1335_pair_three_reductions':
+; LOOPS: Loop at depth 1 containing: %inner.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %outer.header<header>
+
;-------------------------------------------------------------------------------
; (1) Sibling-rich, genuinely shallow nest (true depth 3) whose breadth-first
; descendant count is 12 (top + pairX.outer + pairX.inner + sib1..sib9). The
-; current pass reports the flat count as an unsupported "depth" and never
-; considers the eligible fixed-1335 three-reduction pairX.outer/pairX.inner.
+; unmodified pass reported that flat count as an unsupported "depth" and never
+; considered the eligible fixed-1335 three-reduction pairX.outer/pairX.inner;
+; the depth policy now uses the true nesting depth, so the nest is no longer
+; rejected for depth. Its non-linear list routes directly to the fallback,
+; which interchanges that pair.
;-------------------------------------------------------------------------------
define void @bfs_loop_count_is_not_depth(ptr %A, ptr %B, ptr %C, ptr %R) {
entry:
@@ -350,32 +424,66 @@ exit:
ret void
}
-; The pairX reduction cycle and its address expression are unchanged, and the
-; inner reduction PHIs still take their initial value from %pairX.outer.header
-; (interchange would rewire these incoming edges). The pair remains nested
-; between %top.header and %top.latch, with the sibling chain intact.
+; The pairX candidate pair is interchanged through the fallback. The enclosing
+; top loop and the nine sibling loops are untouched. The three address
+; expressions and reassociated reductions are preserved, and the three checksum
+; results are still stored to %R (observable live-outs). The LOOPS checks below
+; prove the swap and preserved sibling program order.
; IR-LABEL: define void @bfs_loop_count_is_not_depth(
-; IR: %t = phi i64 [ 0, %entry ], [ %t.next, %top.latch ]
-; IR: %sumA.i = phi double [ 0.000000e+00, %top.header ], [ %sumA.i.lcssa, %pairX.outer.latch ]
-; IR: %sumC.i = phi double [ 0.000000e+00, %top.header ], [ %sumC.i.lcssa, %pairX.outer.latch ]
-; IR: %j = phi i64 [ 0, %pairX.outer.header ], [ %j.next, %pairX.inner ]
-; IR: %sumA.j = phi double [ %sumA.i, %pairX.outer.header ], [ %sumA.j.next, %pairX.inner ]
-; IR: %sumC.j = phi double [ %sumC.i, %pairX.outer.header ], [ %sumC.j.next, %pairX.inner ]
-; IR: %idxA = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
-; IR: %sumA.j.next = fadd reassoc double %sumA.j, %a
-; IR: %sumC.j.next = fadd reassoc double %sumC.j, %c
-; IR: %sumA.i.lcssa = phi double [ %sumA.j.next, %pairX.inner ]
-; IR: %sumA.live = phi double [ %sumA.i.lcssa, %pairX.outer.latch ]
-; IR: store double %sumA.live, ptr %R, align 8
-; IR: %s1 = phi i64 [ 0, %pairX.exit ], [ %s1.next, %sib1.header ]
-; IR: %s9 = phi i64 [ 0, %sib8.exit ], [ %s9.next, %sib9.header ]
-; IR: %t.next = add i64 %t, 1
+; IR-DAG: %t = phi i64 [ 0, %entry ], [ %t.next, %top.latch ]
+; IR-DAG: %idxA = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+; IR-DAG: %idxB = getelementptr inbounds [1335 x double], ptr %B, i64 %j, i64 %i
+; IR-DAG: %idxC = getelementptr inbounds [1335 x double], ptr %C, i64 %j, i64 %i
+; IR-DAG: %sumA.j.next = fadd reassoc double %sumA.j, %a
+; IR-DAG: %sumB.j.next = fadd reassoc double %sumB.j, %b
+; IR-DAG: %sumC.j.next = fadd reassoc double %sumC.j, %c
+; IR-DAG: store double %{{.*}}, ptr %R, align 8
+; IR-DAG: store double %{{.*}}, ptr %rB, align 8
+; IR-DAG: store double %{{.*}}, ptr %rC, align 8
+; IR-DAG: %s1 = phi i64 [ 0, %{{.*}} ], [ %s1.next, %sib1.header ]
+; IR-DAG: %s9 = phi i64 [ 0, %{{.*}} ], [ %s9.next, %sib9.header ]
+; IR-DAG: %t.next = add i64 %t, 1
+
+; Swap + sibling-order oracle. The interchanged pair's new outer loop is headed
+; by the former inner header %pairX.inner and its new inner loop by the former
+; outer header %pairX.outer.header (proving a real swap). Crucially, that pair
+; remains the *first* of top's ten children, ahead of the nine untouched sibling
+; loops sib1..sib9 in program order: a LoopInfo update that removed the old child
+; and appended the replacement would push it behind %sib9.header instead.
+; LOOPS-LABEL: Loop info for function 'bfs_loop_count_is_not_depth':
+; LOOPS: Loop at depth 1 containing: %top.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %pairX.inner<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %pairX.outer.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %sib1.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %sib2.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %sib3.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %sib4.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %sib5.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %sib6.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %sib7.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %sib8.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %sib9.header<header>
+;
+; With fallback disabled, the profitable pair keeps its original order.
+; ORIGINAL-NONLINEAR-LOOPS-LABEL: Loop info for function 'bfs_loop_count_is_not_depth':
+; ORIGINAL-NONLINEAR-LOOPS: Loop at depth 1 containing: %top.header<header>
+; ORIGINAL-NONLINEAR-LOOPS-NEXT: Loop at depth 2 containing: %pairX.outer.header<header>
+; ORIGINAL-NONLINEAR-LOOPS-NEXT: Loop at depth 3 containing: %pairX.inner<header>
+; ORIGINAL-NONLINEAR-LOOPS-NEXT: Loop at depth 2 containing: %sib1.header<header>
+; ORIGINAL-NONLINEAR-LOOPS-NEXT: Loop at depth 2 containing: %sib2.header<header>
+; ORIGINAL-NONLINEAR-LOOPS-NEXT: Loop at depth 2 containing: %sib3.header<header>
+; ORIGINAL-NONLINEAR-LOOPS-NEXT: Loop at depth 2 containing: %sib4.header<header>
+; ORIGINAL-NONLINEAR-LOOPS-NEXT: Loop at depth 2 containing: %sib5.header<header>
+; ORIGINAL-NONLINEAR-LOOPS-NEXT: Loop at depth 2 containing: %sib6.header<header>
+; ORIGINAL-NONLINEAR-LOOPS-NEXT: Loop at depth 2 containing: %sib7.header<header>
+; ORIGINAL-NONLINEAR-LOOPS-NEXT: Loop at depth 2 containing: %sib8.header<header>
+; ORIGINAL-NONLINEAR-LOOPS-NEXT: Loop at depth 2 containing: %sib9.header<header>
;-------------------------------------------------------------------------------
; (2) A single SCEV-uncomputable sibling loop (data-dependent exit) sits beside
; a separate, computable, profitable fixed-1335 pair under a common ancestor.
-; isComputableLoopNest rejects the whole flat list, so the pair is not reached
-; and no analysis remark is emitted.
+; The non-linear list routes directly to fallback, which forms the computable
+; pair and interchanges it without querying the unrelated sibling's trip count.
;-------------------------------------------------------------------------------
define void @uncomputable_sibling_does_not_block(ptr %A, ptr %U, ptr %R) {
entry:
@@ -431,24 +539,41 @@ exit:
ret void
}
-; The computable pair keeps its original order; the uncomputable sibling still
-; exits on a loaded value.
+; The computable pair is interchanged through the fallback; the uncomputable
+; sibling is untouched and still exits on a loaded value. The address
+; expression and reassociated reduction are preserved and the result is stored
+; to %R.
; IR-LABEL: define void @uncomputable_sibling_does_not_block(
-; IR: %sum.i = phi double [ 0.000000e+00, %anc.header ], [ %sum.i.lcssa, %pair.outer.latch ]
-; IR: %sum.j = phi double [ %sum.i, %pair.outer.header ], [ %sum.j.next, %pair.inner ]
-; IR: %idx = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
-; IR: %sum.j.next = fadd reassoc double %sum.j, %a
-; IR: %sum.i.lcssa = phi double [ %sum.j.next, %pair.inner ]
-; IR: %sum.live = phi double [ %sum.i.lcssa, %pair.outer.latch ]
-; IR: store double %sum.live, ptr %R, align 8
-; IR: %sv = load double, ptr %sp, align 8
-; IR: %sc = fcmp oeq double %sv, 0.000000e+00
+; IR-DAG: %idx = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+; IR-DAG: %sum.j.next = fadd reassoc double %sum.j, %a
+; IR-DAG: store double %{{.*}}, ptr %R, align 8
+; IR-DAG: %sv = load double, ptr %sp, align 8
+; IR-DAG: %sc = fcmp oeq double %sv, 0.000000e+00
+
+; Swap + sibling-order oracle: the interchanged pair's new outer loop is headed
+; by the former inner header %pair.inner (new inner headed by %pair.outer.header),
+; and it stays ahead of the untouched uncomputable sibling loop %usib.header.
+; LOOPS-LABEL: Loop info for function 'uncomputable_sibling_does_not_block':
+; LOOPS: Loop at depth 1 containing: %anc.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %pair.inner<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %pair.outer.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %usib.header<header>
+;
+; This non-linear nest routes to fallback before any whole-nest analysis
+; remark. With fallback disabled it emits no remark, and the computable pair
+; keeps its original order. The uncomputable sibling is incidental here; the
+; linear uncomputable ancestor below exercises computability rejection.
+; DISABLED-UNCOMPUTABLE-LOOPS-LABEL: Loop info for function 'uncomputable_sibling_does_not_block':
+; DISABLED-UNCOMPUTABLE-LOOPS: Loop at depth 1 containing: %anc.header<header>
+; DISABLED-UNCOMPUTABLE-LOOPS-NEXT: Loop at depth 2 containing: %pair.outer.header<header>
+; DISABLED-UNCOMPUTABLE-LOOPS-NEXT: Loop at depth 3 containing: %pair.inner<header>
+; DISABLED-UNCOMPUTABLE-LOOPS-NEXT: Loop at depth 2 containing: %usib.header<header>
;-------------------------------------------------------------------------------
; (3) A lower, computable fixed-1335 pair beneath an uncomputable *true
; ancestor* (data-dependent latch). The chain is linear, but isComputableLoopNest
-; rejects it because of the ancestor, so the lower pair is never partitioned off
-; and considered.
+; rejects it because of the ancestor, so the standard path never considers the
+; lower pair; the fallback partitions that pair off and interchanges it.
;-------------------------------------------------------------------------------
define void @uncomputable_ancestor_partition(ptr %A, ptr %U, ptr %R) {
entry:
@@ -492,25 +617,31 @@ exit:
ret void
}
-; The pair is untouched and still nested under the uncomputable ancestor.
+; The lower computable pair is interchanged through the fallback even though its
+; true ancestor is uncomputable; the ancestor loop and its data-dependent latch
+; are untouched. The address expression and reassociated reduction are
+; preserved and the result is stored to %R.
; IR-LABEL: define void @uncomputable_ancestor_partition(
-; IR: %k = phi i64 [ 0, %entry ], [ %k.next, %anc.latch ]
-; IR: %sum.i = phi double [ 0.000000e+00, %anc.header ], [ %sum.i.lcssa, %pair.outer.latch ]
-; IR: %sum.j = phi double [ %sum.i, %pair.outer.header ], [ %sum.j.next, %pair.inner ]
-; IR: %idx = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
-; IR: %sum.j.next = fadd reassoc double %sum.j, %a
-; IR: %sum.i.lcssa = phi double [ %sum.j.next, %pair.inner ]
-; IR: %sum.live = phi double [ %sum.i.lcssa, %pair.outer.latch ]
-; IR: store double %sum.live, ptr %R, align 8
-; IR: %kv = load double, ptr %kp, align 8
-; IR: %kc = fcmp oeq double %kv, 0.000000e+00
+; IR-DAG: %k = phi i64 [ 0, %entry ], [ %k.next, %anc.latch ]
+; IR-DAG: %idx = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+; IR-DAG: %sum.j.next = fadd reassoc double %sum.j, %a
+; IR-DAG: store double %{{.*}}, ptr %R, align 8
+; IR-DAG: %kv = load double, ptr %kp, align 8
+; IR-DAG: %kc = fcmp oeq double %kv, 0.000000e+00
+
+; Swap oracle (3-level nest). The uncomputable ancestor %anc.header stays the
+; outermost loop; beneath it the interchanged pair's new outer loop is headed by
+; the former inner header %pair.inner and the new inner loop by the former outer
+; header %pair.outer.header.
+; LOOPS-LABEL: Loop info for function 'uncomputable_ancestor_partition':
+; LOOPS: Loop at depth 1 containing: %anc.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %pair.inner<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %pair.outer.header<header>
;-------------------------------------------------------------------------------
; (4) Two eligible direct single-child pairs (pairA, pairB) share one ancestor,
-; making the breadth-first list non-linear. The current pass emits the analysis
-; remark, then bails at the linearity check, leaving both pairs unchanged. The
-; follow-on candidate-formation implementation, which transforms at most one
-; fallback candidate per nest, will interchange exactly one of them.
+; making the breadth-first list non-linear. It routes directly to fallback,
+; whose one-transform policy interchanges exactly one pair.
;-------------------------------------------------------------------------------
define void @two_candidate_pairs_one_fallback(ptr %A, ptr %B, ptr %R) {
entry:
@@ -582,27 +713,37 @@ exit:
ret void
}
-; Both pairs keep their original order and address expressions.
+; Exactly one pair is interchanged through the fallback. The deterministic first
+; candidate (pairA) is interchanged; the second (pairB) is left in its original
+; inner-reduction order, proving at most one interchange per invocation. Both
+; address expressions and reassociated reductions are preserved and both
+; results are stored.
; IR-LABEL: define void @two_candidate_pairs_one_fallback(
-; IR: %sumA.j = phi double [ %sumA.i, %pairA.outer.header ], [ %sumA.j.next, %pairA.inner ]
-; IR: %idxA = getelementptr inbounds [1335 x double], ptr %A, i64 %jA, i64 %iA
-; IR: %sumA.j.next = fadd reassoc double %sumA.j, %a
-; IR: %sumA.i.lcssa = phi double [ %sumA.j.next, %pairA.inner ]
-; IR: %sumA.live = phi double [ %sumA.i.lcssa, %pairA.outer.latch ]
-; IR: store double %sumA.live, ptr %R, align 8
-; IR: %sumB.j = phi double [ %sumB.i, %pairB.outer.header ], [ %sumB.j.next, %pairB.inner ]
-; IR: %idxB = getelementptr inbounds [1335 x double], ptr %B, i64 %jB, i64 %iB
-; IR: %sumB.j.next = fadd reassoc double %sumB.j, %b
-; IR: %sumB.i.lcssa = phi double [ %sumB.j.next, %pairB.inner ]
-; IR: %sumB.live = phi double [ %sumB.i.lcssa, %pairB.outer.latch ]
-; IR: store double %sumB.live, ptr %rB, align 8
+; IR-DAG: %idxA = getelementptr inbounds [1335 x double], ptr %A, i64 %jA, i64 %iA
+; IR-DAG: %sumA.j.next = fadd reassoc double %sumA.j, %a
+; IR-DAG: store double %{{.*}}, ptr %R, align 8
+; IR-DAG: %sumB.j = phi double [ %sumB.i, %pairB.outer.header ], [ %sumB.j.next, %pairB.inner ]
+; IR-DAG: %idxB = getelementptr inbounds [1335 x double], ptr %B, i64 %jB, i64 %iB
+; IR-DAG: %sumB.j.next = fadd reassoc double %sumB.j, %b
+; IR-DAG: store double %sumB.live, ptr %rB, align 8
+
+; One-transform + deepest/first-selection oracle. pairA (the deterministic first
+; candidate) is swapped -- its new outer loop is headed by the former inner
+; header %pairA.inner -- while pairB is left unswapped: %pairB.outer.header still
+; heads its outer loop with %pairB.inner nested inside. pairA also stays the
+; first of anc's two children (sibling order preserved).
+; LOOPS-LABEL: Loop info for function 'two_candidate_pairs_one_fallback':
+; LOOPS: Loop at depth 1 containing: %anc.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %pairA.inner<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %pairA.outer.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %pairB.outer.header<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %pairB.inner<header>
;-------------------------------------------------------------------------------
; (5a) Lasting negative: a partly-exact reduction set. sumA is reassociable but
; sumB is a strict fadd, so even though this is a plain admissible 2-deep nest
-; the current pass refuses it (UnsupportedPHIOuter). It must stay refused after
-; the follow-on candidate-formation implementation as well -- reassociation is
-; required on every reordered recurrence.
+; the pass refuses it (UnsupportedPHIOuter), and the fallback must refuse it
+; too -- reassociation is required on every reordered recurrence.
;-------------------------------------------------------------------------------
define void @partly_exact_reduction(ptr %A, ptr %B, ptr %R) {
entry:
@@ -660,11 +801,9 @@ exit:
;-------------------------------------------------------------------------------
; (5b) Lasting negative: a dynamic leading dimension (A[j*n + i]). Here it is
-; embedded beside a sibling loop so the current pass bails at the linearity
-; check regardless of profitability -- the point this precommit pins is simply
-; that it is not transformed. Once the follow-on candidate-formation
-; implementation lands, its fallback reaches this pair and default
-; profitability declines the dynamic stride, so it must remain out of scope.
+; embedded beside a sibling loop, so the non-linear nest routes directly to
+; fallback. Default profitability declines the dynamic stride, so it stays out
+; of scope and the nest is not transformed.
;-------------------------------------------------------------------------------
define void @dynamic_leading_dimension_subnest(ptr %A, i64 %n, ptr %U, ptr %R) {
entry:
@@ -739,11 +878,10 @@ exit:
;-------------------------------------------------------------------------------
; (5c) Lasting negative inside a fallback-triggering shape: an all-exact (strict,
; non-reassoc) reduction pair nested under an ancestor k beside a sibling loop, so
-; the flat list is non-linear and the current pass bails at the linearity check
-; after the analysis remark. The follow-on candidate-formation fallback reaches
-; this direct i/j pair but must decline it -- reordering a strict fadd changes
-; the result, so reassoc is required on every reordered recurrence. The result
-; is stored (real live-out).
+; the non-linear flat list routes directly to fallback. The fallback reaches
+; this direct i/j pair but declines it -- reordering a strict fadd changes the
+; result, so reassoc is required on every reordered recurrence. The result is
+; stored (real live-out).
;-------------------------------------------------------------------------------
define void @all_exact_reduction_subnest(ptr %A, ptr %U, ptr %R) {
entry:
@@ -812,13 +950,21 @@ exit:
; IR: %sum.live = phi double [ %sum.i.lcssa, %pair.outer.latch ]
; IR: store double %sum.live, ptr %R, align 8
+; Lasting-negative structural oracle: the strict reduction is declined, so the
+; pair is NOT swapped -- the former outer header %pair.outer.header still heads
+; the depth-2 loop with %pair.inner nested at depth 3, and the sibling remains.
+; LOOPS-LABEL: Loop info for function 'all_exact_reduction_subnest':
+; LOOPS: Loop at depth 1 containing: %anc.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %pair.outer.header<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %pair.inner<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %sib.header<header>
+
;-------------------------------------------------------------------------------
; (5d) Lasting negative inside the same fallback-triggering shape: a partly-exact
; reduction pair (sumA reassociable, sumB a strict fadd) nested under ancestor k
-; beside a sibling, so the current pass again bails at the linearity check. The
-; follow-on candidate-formation fallback reaches the i/j pair but must decline
-; it -- every reordered recurrence must be reassociable and sumB is not. Both
-; results are stored.
+; beside a sibling, so the non-linear nest routes directly to fallback. The
+; fallback reaches the i/j pair but declines it -- every reordered recurrence
+; must be reassociable and sumB is not. Both results are stored.
;-------------------------------------------------------------------------------
define void @partly_exact_reduction_subnest(ptr %A, ptr %B, ptr %U, ptr %R) {
entry:
@@ -902,10 +1048,9 @@ exit:
; (5e) Lasting negative: an unsupported (non-leaf) candidate structure. The
; eligible-looking i/j pair has an inner loop j that is itself NOT a leaf -- it
; encloses two sibling loops m and n -- so j has two children and the flat list
-; is non-linear. The current pass emits the analysis remark and bails at the
-; linearity check. The follow-on candidate-formation fallback initially selects
-; only a leaf inner loop, so this non-leaf candidate must remain skipped. The
-; candidate store is observable.
+; is non-linear and routes directly to fallback. The fallback only selects a
+; leaf inner loop, so this non-leaf candidate is skipped and the nest is
+; unchanged. The candidate store is observable.
;-------------------------------------------------------------------------------
define void @non_leaf_candidate_subnest(ptr %A) {
entry:
@@ -960,3 +1105,201 @@ exit:
; IR: store double 1.000000e+00, ptr %idx, align 8
; IR: %m = phi i64 [ 0, %j.header ], [ %m.next, %m.header ]
; IR: %n = phi i64 [ 0, %m.exit ], [ %n.next, %n.header ]
+
+; Lasting-negative structural oracle: no eligible candidate forms (j is non-leaf),
+; so nothing is swapped -- i outer, j middle, with the two leaf loops m and n.
+; LOOPS-LABEL: Loop info for function 'non_leaf_candidate_subnest':
+; LOOPS: Loop at depth 1 containing: %i.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %j.header<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %m.header<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %n.header<header>
+
+;-------------------------------------------------------------------------------
+; (6) Distinct-depth deepest-first selection. Two eligible fixed-1335 reduction
+; pairs are siblings under a common top loop (so the breadth-first list is
+; non-linear): a shallow pair sA (inner depth 3) and, one level deeper, a pair dB
+; (inner depth 4). The fallback tries the deepest candidate first, so dB is the
+; one interchanged; sA -- equally eligible -- is left untouched because at most
+; one interchange happens per invocation.
+;-------------------------------------------------------------------------------
+define void @distinct_depth_deepest_first(ptr %A, ptr %B, ptr %R) {
+entry:
+ br label %top.header
+
+top.header:
+ %t = phi i64 [ 0, %entry ], [ %t.next, %top.latch ]
+ br label %sA.outer.header
+
+sA.outer.header:
+ %ia = phi i64 [ 0, %top.header ], [ %ia.next, %sA.outer.latch ]
+ %sumA.i = phi double [ 0.000000e+00, %top.header ], [ %sumA.i.lcssa, %sA.outer.latch ]
+ br label %sA.inner
+
+sA.inner:
+ %ja = phi i64 [ 0, %sA.outer.header ], [ %ja.next, %sA.inner ]
+ %sumA.j = phi double [ %sumA.i, %sA.outer.header ], [ %sumA.j.next, %sA.inner ]
+ %idxA = getelementptr inbounds [1335 x double], ptr %A, i64 %ja, i64 %ia
+ %a = load double, ptr %idxA, align 8
+ %sumA.j.next = fadd reassoc double %sumA.j, %a
+ %ja.next = add i64 %ja, 1
+ %ja.ec = icmp eq i64 %ja.next, 1335
+ br i1 %ja.ec, label %sA.outer.latch, label %sA.inner
+
+sA.outer.latch:
+ %sumA.i.lcssa = phi double [ %sumA.j.next, %sA.inner ]
+ %ia.next = add i64 %ia, 1
+ %ia.ec = icmp eq i64 %ia.next, 1335
+ br i1 %ia.ec, label %sA.exit, label %sA.outer.header
+
+sA.exit:
+ %sumA.live = phi double [ %sumA.i.lcssa, %sA.outer.latch ]
+ store double %sumA.live, ptr %R, align 8
+ br label %mid.header
+
+mid.header:
+ %m = phi i64 [ 0, %sA.exit ], [ %m.next, %mid.latch ]
+ br label %dB.outer.header
+
+dB.outer.header:
+ %ib = phi i64 [ 0, %mid.header ], [ %ib.next, %dB.outer.latch ]
+ %sumB.i = phi double [ 0.000000e+00, %mid.header ], [ %sumB.i.lcssa, %dB.outer.latch ]
+ br label %dB.inner
+
+dB.inner:
+ %jb = phi i64 [ 0, %dB.outer.header ], [ %jb.next, %dB.inner ]
+ %sumB.j = phi double [ %sumB.i, %dB.outer.header ], [ %sumB.j.next, %dB.inner ]
+ %idxB = getelementptr inbounds [1335 x double], ptr %B, i64 %jb, i64 %ib
+ %b = load double, ptr %idxB, align 8
+ %sumB.j.next = fadd reassoc double %sumB.j, %b
+ %jb.next = add i64 %jb, 1
+ %jb.ec = icmp eq i64 %jb.next, 1335
+ br i1 %jb.ec, label %dB.outer.latch, label %dB.inner
+
+dB.outer.latch:
+ %sumB.i.lcssa = phi double [ %sumB.j.next, %dB.inner ]
+ %ib.next = add i64 %ib, 1
+ %ib.ec = icmp eq i64 %ib.next, 1335
+ br i1 %ib.ec, label %dB.exit, label %dB.outer.header
+
+dB.exit:
+ %sumB.live = phi double [ %sumB.i.lcssa, %dB.outer.latch ]
+ %rB = getelementptr inbounds double, ptr %R, i64 1
+ store double %sumB.live, ptr %rB, align 8
+ br label %mid.latch
+
+mid.latch:
+ %m.next = add i64 %m, 1
+ %m.ec = icmp eq i64 %m.next, 4
+ br i1 %m.ec, label %top.latch, label %mid.header
+
+top.latch:
+ %t.next = add i64 %t, 1
+ %t.ec = icmp eq i64 %t.next, 4
+ br i1 %t.ec, label %exit, label %top.header
+
+exit:
+ ret void
+}
+
+; The deeper pair dB is interchanged; the shallower pair sA keeps its original
+; inner-reduction order (its inner PHI still seeds from %sA.outer.header). Both
+; address expressions, both reassociated reductions, and both live-out stores
+; survive.
+; IR-LABEL: define void @distinct_depth_deepest_first(
+; IR-DAG: %sumA.j = phi double [ %sumA.i, %sA.outer.header ], [ %sumA.j.next, %sA.inner ]
+; IR-DAG: %idxA = getelementptr inbounds [1335 x double], ptr %A, i64 %ja, i64 %ia
+; IR-DAG: %sumA.j.next = fadd reassoc double %sumA.j, %a
+; IR-DAG: %idxB = getelementptr inbounds [1335 x double], ptr %B, i64 %jb, i64 %ib
+; IR-DAG: %sumB.j.next = fadd reassoc double %sumB.j, %b
+; IR-DAG: store double %sumA.live, ptr %R, align 8
+; IR-DAG: store double %sumB.live, ptr %rB, align 8
+
+; Deepest-first oracle: only the deeper pair dB is swapped (its new outer loop is
+; headed by the former inner header %dB.inner, at depth 3, above the former outer
+; header %dB.outer.header at depth 4), while the shallower pair sA is left
+; unswapped (%sA.outer.header still heads its depth-2 loop above %sA.inner).
+; LOOPS-LABEL: Loop info for function 'distinct_depth_deepest_first':
+; LOOPS: Loop at depth 1 containing: %top.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %sA.outer.header<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %sA.inner<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %mid.header<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %dB.inner<header>
+; LOOPS-NEXT: Loop at depth 4 containing: %dB.outer.header<header>
+
+;-------------------------------------------------------------------------------
+; (7) The selected pair's own structure is unsupported: the candidate inner loop
+; has two exiting blocks (an early exit at j==500 and the normal exit), so it has
+; no unique exit and is not a computable inner pair. It sits beside a sibling loop
+; so the flat list is non-linear and reaches the fallback, which rejects this one
+; candidate with FallbackUnsupportedPair and leaves the nest unchanged.
+;-------------------------------------------------------------------------------
+define void @unsupported_pair_backedge_subnest(ptr %A, ptr %U) {
+entry:
+ br label %anc.header
+
+anc.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %anc.latch ]
+ br label %pair.outer.header
+
+pair.outer.header:
+ %i = phi i64 [ 0, %anc.header ], [ %i.next, %pair.outer.latch ]
+ br label %pair.inner.header
+
+pair.inner.header:
+ %j = phi i64 [ 0, %pair.outer.header ], [ %j.next, %pair.inner.latch ]
+ %idx = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+ store double 1.000000e+00, ptr %idx, align 8
+ %early = icmp eq i64 %j, 500
+ br i1 %early, label %pair.exit.early, label %pair.inner.latch
+
+pair.inner.latch:
+ %j.next = add i64 %j, 1
+ %j.ec = icmp eq i64 %j.next, 1335
+ br i1 %j.ec, label %pair.exit.normal, label %pair.inner.header
+
+pair.exit.early:
+ br label %pair.outer.latch
+
+pair.exit.normal:
+ br label %pair.outer.latch
+
+pair.outer.latch:
+ %i.next = add i64 %i, 1
+ %i.ec = icmp eq i64 %i.next, 1335
+ br i1 %i.ec, label %sib.preheader, label %pair.outer.header
+
+sib.preheader:
+ br label %sib.header
+
+sib.header:
+ %s = phi i64 [ 0, %sib.preheader ], [ %s.next, %sib.header ]
+ %sp = getelementptr inbounds double, ptr %U, i64 %s
+ %sv = load double, ptr %sp, align 8
+ %s.next = add i64 %s, 1
+ %s.ec = icmp eq i64 %s.next, 4
+ br i1 %s.ec, label %sib.exit, label %sib.header
+
+sib.exit:
+ br label %anc.latch
+
+anc.latch:
+ %k.next = add i64 %k, 1
+ %k.ec = icmp eq i64 %k.next, 4
+ br i1 %k.ec, label %exit, label %anc.header
+
+exit:
+ ret void
+}
+
+; The pair keeps its original order, its two-exit inner loop, and the store; the
+; sibling is untouched. Nothing is interchanged (FallbackUnsupportedPair).
+; IR-LABEL: define void @unsupported_pair_backedge_subnest(
+; IR: %i = phi i64 [ 0, %anc.header ], [ %i.next, %pair.outer.latch ]
+; IR: %j = phi i64 [ 0, %pair.outer.header ], [ %j.next, %pair.inner.latch ]
+; IR: %idx = getelementptr inbounds [1335 x double], ptr %A, i64 %j, i64 %i
+; IR: br i1 %early, label %pair.exit.early, label %pair.inner.latch
+; LOOPS-LABEL: Loop info for function 'unsupported_pair_backedge_subnest':
+; LOOPS: Loop at depth 1 containing: %anc.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %pair.outer.header<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %pair.inner.header<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %sib.header<header>
diff --git a/llvm/test/Transforms/LoopInterchange/inner-subnest-dependences.ll b/llvm/test/Transforms/LoopInterchange/inner-subnest-dependences.ll
index e45e6266ab44a..b8d6d61086c99 100644
--- a/llvm/test/Transforms/LoopInterchange/inner-subnest-dependences.ll
+++ b/llvm/test/Transforms/LoopInterchange/inner-subnest-dependences.ll
@@ -1,4 +1,4 @@
-; Precommit test for the surrounding dependence context of an inner subnest
+; Behavior test for the surrounding dependence context of an inner-subnest
; candidate in LoopInterchange.
;
; This file is entirely hand-maintained. Do NOT run update_test_checks.py on it.
@@ -6,20 +6,26 @@
; Every function has a true ancestor loop `k`, an adjacent candidate pair
; `i`(outer)/`j`(inner), and a *sibling* loop `s` with its own memory traffic.
; Because the ancestor has two child loops the breadth-first LoopNest list is
-; not a single linear chain. LoopInterchangePass::run still emits the analysis
-; remark (the depth and computability checks pass), and then
-; LoopInterchange::run(LoopNest&) bails at its linearity check -- nothing is
-; interchanged today. That is all this precommit pins: the analysis remark plus
-; the actual unswapped IR (original k/i/j nesting, address expressions, and the
-; sibling's separate memory operations left in place).
+; not a single linear chain, so LoopInterchangePass::run routes it to the
+; inner-subnest fallback. The fallback builds the candidate's direction matrix
+; over the full k/i/j ancestor chain, collecting memory only from the candidate
+; outer loop (so the sibling `s` traffic is excluded), and applies the
+; conservative ancestor-prefix rule.
;
-; The four access patterns set up the ancestor-prefix cases that the follow-on
-; candidate-formation implementation will distinguish (known-forward,
-; equal/legal, unknown, equal/unsafe), and the last function makes the sibling
-; store overlap the candidate array to pin that the follow-on implementation
-; must NOT fold sibling memory into the candidate's direction matrix. This
-; precommit deliberately does not assert those legality outcomes or direction
-; vectors; the unmodified pass never computes the candidate pair's matrix here.
+; Four access patterns exercise the ancestor-prefix cases: known-forward and
+; equal/legal are legal to interchange, while unknown and equal/unsafe are
+; rejected. `sibling_store_not_a_candidate_dimension` overlaps sibling traffic
+; with the candidate array and covers the excluded-sibling placement.
+; Its contrast, `folded_store_in_candidate_rejects`, places the extra store
+; inside the candidate and verifies that candidate memory is collected.
+;
+; Under default profitability the candidate arrays are indexed with `j`
+; innermost (unit stride), so interchange is legal-but-unprofitable and the IR
+; is unchanged; the IR run pins that. The LEGAL run isolates legality from
+; profitability with -loop-interchange-profitabilities=ignore: the two legal
+; prefixes and the sibling-overlap case are interchanged, while the
+; unknown-ancestor, unsafe-column, and in-candidate contrast cases are rejected.
+; The DA run pins that the unknown-ancestor dependence is real (not confused).
;
; RUN: opt < %s -passes=loop-interchange -cache-line-size=64 \
; RUN: -verify-dom-info -verify-loop-info -verify-scev -verify-loop-lcssa \
@@ -28,15 +34,32 @@
; RUN: opt < %s -passes=loop-interchange -cache-line-size=64 \
; RUN: -pass-remarks=loop-interchange -pass-remarks-missed=loop-interchange \
; RUN: -pass-remarks-output=%t -disable-output
-; RUN: FileCheck %s --check-prefix=YAML --input-file=%t
+; RUN: FileCheck %s --check-prefix=YAML --input-file=%t \
+; RUN: --implicit-check-not='Computed dependence info'
+;
+; Legality isolated from profitability. The two legal prefixes and the
+; sibling-overlap case reach Passed selection with candidate matrices that
+; exclude the sibling. The unknown ancestor, unsafe candidate, and in-candidate
+; contrast cases are rejected. The analysis verifiers must pass after each
+; selected transform.
+; RUN: opt < %s -passes=loop-interchange -cache-line-size=64 \
+; RUN: -loop-interchange-profitabilities=ignore \
+; RUN: -verify-dom-info -verify-loop-info -verify-scev -verify-loop-lcssa \
+; RUN: -pass-remarks=loop-interchange -pass-remarks-missed=loop-interchange \
+; RUN: -pass-remarks-output=%t.ignore -disable-output
+; RUN: FileCheck %s --check-prefix=LEGAL --input-file=%t.ignore
;
; The unknown-ancestor fixture must expose a *real*, non-confused surrounding
; dependence whose ancestor (k) column is unknown while the selected i/j columns
-; are known and legal in isolation. Prove that on the unmodified pass with the
-; dependence-analysis printer: DA reports a genuine flow/anti dependence whose
-; outermost (k) direction is `*` -- not `confused!`. (LoopInterchange's own
-; matrix normalizes this exact CF[j-1][i] = CF[j][i] shape to `* = <`; see the
-; all_eq_lt case in legality-check.ll.)
+; are known and legal in isolation. Prove that independently of LoopInterchange
+; with the dependence-analysis printer: DA reports a genuine flow/anti
+; dependence whose outermost (k) direction is `*` -- not `confused!`.
+; (LoopInterchange's own matrix normalizes this exact CF[j-1][i] = CF[j][i]
+; shape to `* = <`; see the all_eq_lt case in legality-check.ll.) The fallback's
+; conservative prefix rule rejects this `* = <` because the ancestor direction
+; is unknown. Dropping the ancestor would expose only the legal `[= <]` i/j
+; columns and interchange the pair, matching the standard path's policy for
+; this matrix; this fixture pins the fallback's stricter decision.
; RUN: opt < %s -passes='print<da>' -aa-pipeline=basic-aa -disable-output 2>&1 \
; RUN: | FileCheck %s --check-prefix=DA
@@ -48,30 +71,64 @@ target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
@CF = global [8 x [8 x double]] zeroinitializer
@SB = global [8 x [8 x double]] zeroinitializer
-; Each nest reaches the transform (analysis remark) but is then rejected as
-; non-linear; none is interchanged.
-; YAML: --- !Analysis
-; YAML: Name: Dependence
+; Default profitability: each nest reaches the inner-subnest fallback. The two
+; legal prefixes and the sibling-overlap case pass legality but are declined by
+; profitability (unit-stride `j`); the unknown ancestor prefix and the unsafe
+; candidate columns are rejected before profitability.
+; YAML: --- !Missed
+; YAML: Name: InterchangeNotProfitable
; YAML: Function: dep_known_forward_ancestor
-; YAML: --- !Analysis
-; YAML: Name: Dependence
+; YAML: --- !Missed
+; YAML: Name: InterchangeNotProfitable
; YAML: Function: dep_equal_legal_ancestor
-; YAML: --- !Analysis
-; YAML: Name: Dependence
+; The unknown ancestor prefix is rejected by the conservative prefix rule.
+; YAML: --- !Missed
+; YAML: Name: FallbackUnknownContext
; YAML: Function: dep_unknown_ancestor
-; YAML: --- !Analysis
+; The unsafe candidate columns are rejected by the existing pair legality.
+; YAML: --- !Missed
; YAML: Name: Dependence
; YAML: Function: dep_equal_unsafe_ancestor
-; YAML: --- !Analysis
-; YAML: Name: Dependence
+; YAML: --- !Missed
+; YAML: Name: InterchangeNotProfitable
; YAML: Function: sibling_store_not_a_candidate_dimension
+; The contrast fixture: the extra store is inside the candidate loop, so it is
+; collected and rejects the swap on dependence grounds (not profitability).
+; YAML: --- !Missed
+; YAML: Name: Dependence
+; YAML: Function: folded_store_in_candidate_rejects
+
+; With profitability ignored, legality alone decides. The known-forward and
+; equal/legal prefixes interchange; the sibling-overlap case interchanges too,
+; covering overlapping traffic outside the candidate matrix. The unknown
+; ancestor prefix and the unsafe candidate columns are still rejected.
+; LEGAL: --- !Passed
+; LEGAL: Name: Interchanged
+; LEGAL: Function: dep_known_forward_ancestor
+; LEGAL: --- !Passed
+; LEGAL: Name: Interchanged
+; LEGAL: Function: dep_equal_legal_ancestor
+; LEGAL: --- !Missed
+; LEGAL: Name: FallbackUnknownContext
+; LEGAL: Function: dep_unknown_ancestor
+; LEGAL: --- !Missed
+; LEGAL: Name: Dependence
+; LEGAL: Function: dep_equal_unsafe_ancestor
+; LEGAL: --- !Passed
+; LEGAL: Name: Interchanged
+; LEGAL: Function: sibling_store_not_a_candidate_dimension
+; The contrast fixture rejects even with profitability ignored: legality alone
+; declines it because the in-candidate j-invariant store carries `*` in j.
+; LEGAL: --- !Missed
+; LEGAL: Name: Dependence
+; LEGAL: Function: folded_store_in_candidate_rejects
+; LEGAL-NOT: Function: folded_store_in_candidate_rejects
;-------------------------------------------------------------------------------
; Known-forward ancestor prefix: the store to KF[k+1][i][j] is read at KF[k][i][j]
; on the next k iteration, a lexicographically forward carry on the ancestor.
-; That forward prefix is decisive for the follow-on candidate-formation
-; implementation, which will find the i/j swap legal; here the nest is simply
-; not processed (non-linear).
+; That forward prefix is decisive, so the i/j swap is legal (LEGAL run); under
+; default profitability it is unit-stride and therefore declined.
;-------------------------------------------------------------------------------
define void @dep_known_forward_ancestor() {
entry:
@@ -138,9 +195,8 @@ exit:
;-------------------------------------------------------------------------------
; Equal/legal ancestor prefix: a read-modify-write of the same EQ[k][i][j], a
-; loop-independent (equal) dependence. For the follow-on candidate-formation
-; implementation the equal ancestor prefix delegates to the i/j columns, which
-; are legal to swap.
+; loop-independent (equal) dependence. The equal ancestor prefix delegates to
+; the i/j columns, which are legal to swap (LEGAL run).
;-------------------------------------------------------------------------------
define void @dep_equal_legal_ancestor() {
entry:
@@ -205,12 +261,11 @@ exit:
; the selected i/j columns are perfectly known -- equal in i and unit-carried in
; j (CF[j-1][i] = CF[j][i], the all_eq_lt shape from legality-check.ll whose
; interchange matrix is `* = <`). Dependence analysis returns a real dependence,
-; not `confused!` (see the DA run). The strict fallback in the follow-on
-; candidate-formation implementation must reject this pair because the true
-; ancestor context is unknown; an unsound implementation that dropped or
-; projected the ancestor away would see only the legal `[= <]` i/j columns and
-; wrongly accept. This precommit only pins that the dependence is real and that
-; nothing is interchanged.
+; not `confused!` (see the DA run). The fallback's conservative prefix rule
+; rejects this pair (FallbackUnknownContext) because the ancestor direction is
+; unknown. Dropping the ancestor would expose only the legal `[= <]` i/j
+; columns and interchange the pair, as the standard path does for this matrix;
+; this fixture pins the fallback's stricter decision.
;-------------------------------------------------------------------------------
define void @dep_unknown_ancestor() {
entry:
@@ -285,8 +340,8 @@ exit:
;-------------------------------------------------------------------------------
; Equal ancestor prefix, unsafe selected columns: US[k][i][j+1] is written from
; US[k][i+1][j], a cross i/j carry that is not safe to interchange even though
-; the ancestor prefix is equal. The i/j columns themselves must reject in the
-; follow-on candidate-formation implementation.
+; the ancestor prefix is equal. The i/j columns themselves are rejected by the
+; existing pair legality (LEGAL run).
;-------------------------------------------------------------------------------
define void @dep_equal_unsafe_ancestor() {
entry:
@@ -349,12 +404,16 @@ exit:
;-------------------------------------------------------------------------------
; The sibling loop stores into the *same* array KF that the candidate reads, but
-; on a different (diagonal) index. If the follow-on candidate-formation
-; implementation ever collected the candidate pair's direction matrix from the
-; whole ancestor subtree it would pull in this sibling store and mis-model it as
-; a candidate dimension. Today the nest is simply non-linear and untouched; this
-; precommit pins that both the candidate access and the sibling store are
-; present and unchanged.
+; on a different (diagonal) index. The fallback collects the candidate matrix
+; only from the candidate outer loop, so this sibling store is excluded and the
+; candidate interchanges with profitability ignored (LEGAL run); under default
+; profitability it is unit-stride and unchanged (IR run).
+;
+; The contrast fixture folded_store_in_candidate_rejects places an extra store
+; inside the candidate j loop, where it is collected. That store's j-invariant
+; self-output dependence carries `*` in the j column and defeats interchange.
+; The two fixtures cover sibling and in-candidate placements with different
+; access patterns; they do not assert an if-and-only-if relationship.
;-------------------------------------------------------------------------------
define void @sibling_store_not_a_candidate_dimension() {
entry:
@@ -412,3 +471,75 @@ exit:
; IR: store double %nv, ptr %idx, align 8
; IR: %sib.idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @KF, i64 0, i64 %k, i64 %s, i64 %s
; IR: store double 1.000000e+00, ptr %sib.idx, align 8
+
+;-------------------------------------------------------------------------------
+; Contrast fixture for the sibling-exclusion claim above. The candidate does the
+; same KF[k][i][j] read-modify-write, but an extra store to the j-invariant
+; address KF[k][i][0] now sits *inside* the candidate j loop (not in a sibling).
+; Because it is inside the candidate outer loop, it is collected into the matrix,
+; and its output self-dependence carries `*` in the j column (the same location
+; is written on every j), so the i/j swap is rejected on dependence grounds --
+; even with profitability ignored. This covers the in-candidate placement.
+; Together with the sibling fixture it covers both placements, but the two
+; fixtures use different access patterns and do not establish an if-and-only-if
+; relationship.
+;-------------------------------------------------------------------------------
+define void @folded_store_in_candidate_rejects() {
+entry:
+ br label %k.header
+
+k.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %k.latch ]
+ br label %i.header
+
+i.header:
+ %i = phi i64 [ 0, %k.header ], [ %i.next, %i.latch ]
+ br label %j.body
+
+j.body:
+ %j = phi i64 [ 0, %i.header ], [ %j.next, %j.body ]
+ %idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @KF, i64 0, i64 %k, i64 %i, i64 %j
+ %v = load double, ptr %idx, align 8
+ %nv = fadd double %v, 1.000000e+00
+ store double %nv, ptr %idx, align 8
+ %inv.idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @KF, i64 0, i64 %k, i64 %i, i64 0
+ store double %nv, ptr %inv.idx, align 8
+ %j.next = add i64 %j, 1
+ %j.ec = icmp eq i64 %j.next, 7
+ br i1 %j.ec, label %i.latch, label %j.body
+
+i.latch:
+ %i.next = add i64 %i, 1
+ %i.ec = icmp eq i64 %i.next, 7
+ br i1 %i.ec, label %sib.preheader, label %i.header
+
+sib.preheader:
+ br label %sib.header
+
+sib.header:
+ %s = phi i64 [ 0, %sib.preheader ], [ %s.next, %sib.header ]
+ %s.idx = getelementptr inbounds [8 x [8 x double]], ptr @SB, i64 0, i64 %k, i64 %s
+ %s.v = load double, ptr %s.idx, align 8
+ %s.nv = fadd double %s.v, 1.000000e+00
+ store double %s.nv, ptr %s.idx, align 8
+ %s.next = add i64 %s, 1
+ %s.ec = icmp eq i64 %s.next, 7
+ br i1 %s.ec, label %k.latch, label %sib.header
+
+k.latch:
+ %k.next = add i64 %k, 1
+ %k.ec = icmp eq i64 %k.next, 7
+ br i1 %k.ec, label %exit, label %k.header
+
+exit:
+ ret void
+}
+
+; The candidate keeps its original i/j order: the extra j-invariant store
+; KF[k][i][0] inside the j loop is collected and its `*`-in-j self output
+; dependence declines the swap (contrast with the excluded sibling above).
+; IR-LABEL: define void @folded_store_in_candidate_rejects(
+; IR: %i = phi i64 [ 0, %k.header ], [ %i.next, %i.latch ]
+; IR: %j = phi i64 [ 0, %i.header ], [ %j.next, %j.body ]
+; IR: %idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @KF, i64 0, i64 %k, i64 %i, i64 %j
+; IR: %inv.idx = getelementptr inbounds [8 x [8 x [8 x double]]], ptr @KF, i64 0, i64 %k, i64 %i, i64 0
diff --git a/llvm/test/Transforms/LoopInterchange/inner-subnest-enumeration.ll b/llvm/test/Transforms/LoopInterchange/inner-subnest-enumeration.ll
new file mode 100644
index 0000000000000..60678af42be26
--- /dev/null
+++ b/llvm/test/Transforms/LoopInterchange/inner-subnest-enumeration.ll
@@ -0,0 +1,315 @@
+; Exercise candidate enumeration on a deep spine ending in a wide set of
+; sibling outer/leaf pairs. Pinning the supported depth to the candidates'
+; actual depth while setting the attempt budget to seven requires all eight
+; candidates to be enumerated. The first seven are attempted in breadth-first
+; order, and the eighth anchors the budget remark.
+;
+; RUN: opt < %s -passes=loop-interchange \
+; RUN: -loop-interchange-min-loop-nest-depth=10 \
+; RUN: -loop-interchange-max-loop-nest-depth=10 \
+; RUN: -loop-interchange-max-inner-subnest-candidates=7 \
+; RUN: -verify-dom-info -verify-loop-info -verify-scev -verify-loop-lcssa \
+; RUN: -pass-remarks-output=%t.yaml -disable-output
+; RUN: FileCheck %s --input-file=%t.yaml --implicit-check-not=Interchanged
+; RUN: opt < %s -passes='loop(loop-interchange),print<loops>' \
+; RUN: -loop-interchange-min-loop-nest-depth=10 \
+; RUN: -loop-interchange-max-loop-nest-depth=10 \
+; RUN: -loop-interchange-max-inner-subnest-candidates=7 \
+; RUN: -disable-output 2>&1 | FileCheck %s --check-prefix=LOOPS
+
+; CHECK-NOT: --- !
+; CHECK: --- !Missed
+; CHECK-NEXT: Pass: loop-interchange
+; CHECK-NEXT: Name: InterchangeNotProfitable
+; CHECK-NEXT: DebugLoc: { File: inner-subnest-enumeration.ll, Line: 10, Column: 1 }
+; CHECK-NEXT: Function: deep_spine_wide_siblings
+; CHECK-NEXT: Args:
+; CHECK-NEXT: - String: Insufficient information to calculate the cost of loop for interchange.
+; CHECK-NEXT: ...
+; CHECK-NEXT: --- !Missed
+; CHECK-NEXT: Pass: loop-interchange
+; CHECK-NEXT: Name: InterchangeNotProfitable
+; CHECK-NEXT: DebugLoc: { File: inner-subnest-enumeration.ll, Line: 20, Column: 1 }
+; CHECK-NEXT: Function: deep_spine_wide_siblings
+; CHECK-NEXT: Args:
+; CHECK-NEXT: - String: Insufficient information to calculate the cost of loop for interchange.
+; CHECK-NEXT: ...
+; CHECK-NEXT: --- !Missed
+; CHECK-NEXT: Pass: loop-interchange
+; CHECK-NEXT: Name: InterchangeNotProfitable
+; CHECK-NEXT: DebugLoc: { File: inner-subnest-enumeration.ll, Line: 30, Column: 1 }
+; CHECK-NEXT: Function: deep_spine_wide_siblings
+; CHECK-NEXT: Args:
+; CHECK-NEXT: - String: Insufficient information to calculate the cost of loop for interchange.
+; CHECK-NEXT: ...
+; CHECK-NEXT: --- !Missed
+; CHECK-NEXT: Pass: loop-interchange
+; CHECK-NEXT: Name: InterchangeNotProfitable
+; CHECK-NEXT: DebugLoc: { File: inner-subnest-enumeration.ll, Line: 40, Column: 1 }
+; CHECK-NEXT: Function: deep_spine_wide_siblings
+; CHECK-NEXT: Args:
+; CHECK-NEXT: - String: Insufficient information to calculate the cost of loop for interchange.
+; CHECK-NEXT: ...
+; CHECK-NEXT: --- !Missed
+; CHECK-NEXT: Pass: loop-interchange
+; CHECK-NEXT: Name: InterchangeNotProfitable
+; CHECK-NEXT: DebugLoc: { File: inner-subnest-enumeration.ll, Line: 50, Column: 1 }
+; CHECK-NEXT: Function: deep_spine_wide_siblings
+; CHECK-NEXT: Args:
+; CHECK-NEXT: - String: Insufficient information to calculate the cost of loop for interchange.
+; CHECK-NEXT: ...
+; CHECK-NEXT: --- !Missed
+; CHECK-NEXT: Pass: loop-interchange
+; CHECK-NEXT: Name: InterchangeNotProfitable
+; CHECK-NEXT: DebugLoc: { File: inner-subnest-enumeration.ll, Line: 60, Column: 1 }
+; CHECK-NEXT: Function: deep_spine_wide_siblings
+; CHECK-NEXT: Args:
+; CHECK-NEXT: - String: Insufficient information to calculate the cost of loop for interchange.
+; CHECK-NEXT: ...
+; CHECK-NEXT: --- !Missed
+; CHECK-NEXT: Pass: loop-interchange
+; CHECK-NEXT: Name: InterchangeNotProfitable
+; CHECK-NEXT: DebugLoc: { File: inner-subnest-enumeration.ll, Line: 70, Column: 1 }
+; CHECK-NEXT: Function: deep_spine_wide_siblings
+; CHECK-NEXT: Args:
+; CHECK-NEXT: - String: Insufficient information to calculate the cost of loop for interchange.
+; CHECK-NEXT: ...
+; CHECK-NEXT: --- !Missed
+; CHECK-NEXT: Pass: loop-interchange
+; CHECK-NEXT: Name: FallbackCandidateBudget
+; CHECK-NEXT: DebugLoc: { File: inner-subnest-enumeration.ll, Line: 80, Column: 1 }
+; CHECK-NEXT: Function: deep_spine_wide_siblings
+; CHECK-NEXT: Args:
+; CHECK-NEXT: - String: 'Inner-subnest candidate budget exhausted; the loop nest is left unchanged.'
+; CHECK-NEXT: ...
+; CHECK-NOT: --- !
+
+; LOOPS-LABEL: Loop info for function 'deep_spine_wide_siblings':
+; LOOPS: Loop at depth 1 containing: %d1.h<header>
+; LOOPS-NEXT: Loop at depth 2 containing: %d2.h<header>
+; LOOPS-NEXT: Loop at depth 3 containing: %d3.h<header>
+; LOOPS-NEXT: Loop at depth 4 containing: %d4.h<header>
+; LOOPS-NEXT: Loop at depth 5 containing: %d5.h<header>
+; LOOPS-NEXT: Loop at depth 6 containing: %d6.h<header>
+; LOOPS-NEXT: Loop at depth 7 containing: %d7.h<header>
+; LOOPS-NEXT: Loop at depth 8 containing: %d8.h<header>
+; LOOPS-NEXT: Loop at depth 9 containing: %p0.o.h<header>
+; LOOPS-NEXT: Loop at depth 10 containing: %p0.i.h<header>
+; LOOPS-NEXT: Loop at depth 9 containing: %p1.o.h<header>
+; LOOPS-NEXT: Loop at depth 10 containing: %p1.i.h<header>
+; LOOPS-NEXT: Loop at depth 9 containing: %p2.o.h<header>
+; LOOPS-NEXT: Loop at depth 10 containing: %p2.i.h<header>
+; LOOPS-NEXT: Loop at depth 9 containing: %p3.o.h<header>
+; LOOPS-NEXT: Loop at depth 10 containing: %p3.i.h<header>
+; LOOPS-NEXT: Loop at depth 9 containing: %p4.o.h<header>
+; LOOPS-NEXT: Loop at depth 10 containing: %p4.i.h<header>
+; LOOPS-NEXT: Loop at depth 9 containing: %p5.o.h<header>
+; LOOPS-NEXT: Loop at depth 10 containing: %p5.i.h<header>
+; LOOPS-NEXT: Loop at depth 9 containing: %p6.o.h<header>
+; LOOPS-NEXT: Loop at depth 10 containing: %p6.i.h<header>
+; LOOPS-NEXT: Loop at depth 9 containing: %p7.o.h<header>
+; LOOPS-NEXT: Loop at depth 10 containing: %p7.i.h<header>
+
+define void @deep_spine_wide_siblings() !dbg !5 {
+entry:
+ br label %d1.h
+
+d1.h:
+ %d1 = phi i64 [ 0, %entry ], [ %d1.n, %d1.l ]
+ br label %d2.h
+d2.h:
+ %d2 = phi i64 [ 0, %d1.h ], [ %d2.n, %d2.l ]
+ br label %d3.h
+d3.h:
+ %d3 = phi i64 [ 0, %d2.h ], [ %d3.n, %d3.l ]
+ br label %d4.h
+d4.h:
+ %d4 = phi i64 [ 0, %d3.h ], [ %d4.n, %d4.l ]
+ br label %d5.h
+d5.h:
+ %d5 = phi i64 [ 0, %d4.h ], [ %d5.n, %d5.l ]
+ br label %d6.h
+d6.h:
+ %d6 = phi i64 [ 0, %d5.h ], [ %d6.n, %d6.l ]
+ br label %d7.h
+d7.h:
+ %d7 = phi i64 [ 0, %d6.h ], [ %d7.n, %d7.l ]
+ br label %d8.h
+d8.h:
+ %d8 = phi i64 [ 0, %d7.h ], [ %d8.n, %d8.l ]
+ br label %p0.o.h
+
+p0.o.h:
+ %p0.o = phi i64 [ 0, %d8.h ], [ %p0.o.n, %p0.o.l ]
+ br label %p0.i.h, !dbg !6
+p0.i.h:
+ %p0.i = phi i64 [ 0, %p0.o.h ], [ %p0.i.n, %p0.i.h ]
+ %p0.i.n = add i64 %p0.i, 1
+ %p0.i.e = icmp eq i64 %p0.i.n, 2
+ br i1 %p0.i.e, label %p0.o.l, label %p0.i.h
+p0.o.l:
+ %p0.o.n = add i64 %p0.o, 1
+ %p0.o.e = icmp eq i64 %p0.o.n, 2
+ br i1 %p0.o.e, label %p0.exit, label %p0.o.h
+p0.exit:
+ br label %p1.o.h
+
+p1.o.h:
+ %p1.o = phi i64 [ 0, %p0.exit ], [ %p1.o.n, %p1.o.l ]
+ br label %p1.i.h, !dbg !7
+p1.i.h:
+ %p1.i = phi i64 [ 0, %p1.o.h ], [ %p1.i.n, %p1.i.h ]
+ %p1.i.n = add i64 %p1.i, 1
+ %p1.i.e = icmp eq i64 %p1.i.n, 2
+ br i1 %p1.i.e, label %p1.o.l, label %p1.i.h
+p1.o.l:
+ %p1.o.n = add i64 %p1.o, 1
+ %p1.o.e = icmp eq i64 %p1.o.n, 2
+ br i1 %p1.o.e, label %p1.exit, label %p1.o.h
+p1.exit:
+ br label %p2.o.h
+
+p2.o.h:
+ %p2.o = phi i64 [ 0, %p1.exit ], [ %p2.o.n, %p2.o.l ]
+ br label %p2.i.h, !dbg !8
+p2.i.h:
+ %p2.i = phi i64 [ 0, %p2.o.h ], [ %p2.i.n, %p2.i.h ]
+ %p2.i.n = add i64 %p2.i, 1
+ %p2.i.e = icmp eq i64 %p2.i.n, 2
+ br i1 %p2.i.e, label %p2.o.l, label %p2.i.h
+p2.o.l:
+ %p2.o.n = add i64 %p2.o, 1
+ %p2.o.e = icmp eq i64 %p2.o.n, 2
+ br i1 %p2.o.e, label %p2.exit, label %p2.o.h
+p2.exit:
+ br label %p3.o.h
+
+p3.o.h:
+ %p3.o = phi i64 [ 0, %p2.exit ], [ %p3.o.n, %p3.o.l ]
+ br label %p3.i.h, !dbg !9
+p3.i.h:
+ %p3.i = phi i64 [ 0, %p3.o.h ], [ %p3.i.n, %p3.i.h ]
+ %p3.i.n = add i64 %p3.i, 1
+ %p3.i.e = icmp eq i64 %p3.i.n, 2
+ br i1 %p3.i.e, label %p3.o.l, label %p3.i.h
+p3.o.l:
+ %p3.o.n = add i64 %p3.o, 1
+ %p3.o.e = icmp eq i64 %p3.o.n, 2
+ br i1 %p3.o.e, label %p3.exit, label %p3.o.h
+p3.exit:
+ br label %p4.o.h
+
+p4.o.h:
+ %p4.o = phi i64 [ 0, %p3.exit ], [ %p4.o.n, %p4.o.l ]
+ br label %p4.i.h, !dbg !10
+p4.i.h:
+ %p4.i = phi i64 [ 0, %p4.o.h ], [ %p4.i.n, %p4.i.h ]
+ %p4.i.n = add i64 %p4.i, 1
+ %p4.i.e = icmp eq i64 %p4.i.n, 2
+ br i1 %p4.i.e, label %p4.o.l, label %p4.i.h
+p4.o.l:
+ %p4.o.n = add i64 %p4.o, 1
+ %p4.o.e = icmp eq i64 %p4.o.n, 2
+ br i1 %p4.o.e, label %p4.exit, label %p4.o.h
+p4.exit:
+ br label %p5.o.h
+
+p5.o.h:
+ %p5.o = phi i64 [ 0, %p4.exit ], [ %p5.o.n, %p5.o.l ]
+ br label %p5.i.h, !dbg !11
+p5.i.h:
+ %p5.i = phi i64 [ 0, %p5.o.h ], [ %p5.i.n, %p5.i.h ]
+ %p5.i.n = add i64 %p5.i, 1
+ %p5.i.e = icmp eq i64 %p5.i.n, 2
+ br i1 %p5.i.e, label %p5.o.l, label %p5.i.h
+p5.o.l:
+ %p5.o.n = add i64 %p5.o, 1
+ %p5.o.e = icmp eq i64 %p5.o.n, 2
+ br i1 %p5.o.e, label %p5.exit, label %p5.o.h
+p5.exit:
+ br label %p6.o.h
+
+p6.o.h:
+ %p6.o = phi i64 [ 0, %p5.exit ], [ %p6.o.n, %p6.o.l ]
+ br label %p6.i.h, !dbg !12
+p6.i.h:
+ %p6.i = phi i64 [ 0, %p6.o.h ], [ %p6.i.n, %p6.i.h ]
+ %p6.i.n = add i64 %p6.i, 1
+ %p6.i.e = icmp eq i64 %p6.i.n, 2
+ br i1 %p6.i.e, label %p6.o.l, label %p6.i.h
+p6.o.l:
+ %p6.o.n = add i64 %p6.o, 1
+ %p6.o.e = icmp eq i64 %p6.o.n, 2
+ br i1 %p6.o.e, label %p6.exit, label %p6.o.h
+p6.exit:
+ br label %p7.o.h
+
+p7.o.h:
+ %p7.o = phi i64 [ 0, %p6.exit ], [ %p7.o.n, %p7.o.l ]
+ br label %p7.i.h, !dbg !13
+p7.i.h:
+ %p7.i = phi i64 [ 0, %p7.o.h ], [ %p7.i.n, %p7.i.h ]
+ %p7.i.n = add i64 %p7.i, 1
+ %p7.i.e = icmp eq i64 %p7.i.n, 2
+ br i1 %p7.i.e, label %p7.o.l, label %p7.i.h
+p7.o.l:
+ %p7.o.n = add i64 %p7.o, 1
+ %p7.o.e = icmp eq i64 %p7.o.n, 2
+ br i1 %p7.o.e, label %p7.exit, label %p7.o.h
+p7.exit:
+ br label %d8.l
+
+d8.l:
+ %d8.n = add i64 %d8, 1
+ %d8.e = icmp eq i64 %d8.n, 2
+ br i1 %d8.e, label %d7.l, label %d8.h
+d7.l:
+ %d7.n = add i64 %d7, 1
+ %d7.e = icmp eq i64 %d7.n, 2
+ br i1 %d7.e, label %d6.l, label %d7.h
+d6.l:
+ %d6.n = add i64 %d6, 1
+ %d6.e = icmp eq i64 %d6.n, 2
+ br i1 %d6.e, label %d5.l, label %d6.h
+d5.l:
+ %d5.n = add i64 %d5, 1
+ %d5.e = icmp eq i64 %d5.n, 2
+ br i1 %d5.e, label %d4.l, label %d5.h
+d4.l:
+ %d4.n = add i64 %d4, 1
+ %d4.e = icmp eq i64 %d4.n, 2
+ br i1 %d4.e, label %d3.l, label %d4.h
+d3.l:
+ %d3.n = add i64 %d3, 1
+ %d3.e = icmp eq i64 %d3.n, 2
+ br i1 %d3.e, label %d2.l, label %d3.h
+d2.l:
+ %d2.n = add i64 %d2, 1
+ %d2.e = icmp eq i64 %d2.n, 2
+ br i1 %d2.e, label %d1.l, label %d2.h
+d1.l:
+ %d1.n = add i64 %d1, 1
+ %d1.e = icmp eq i64 %d1.n, 2
+ br i1 %d1.e, label %exit, label %d1.h
+
+exit:
+ ret void
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "llvm", isOptimized: true, runtimeVersion: 0, emissionKind: LineTablesOnly)
+!1 = !DIFile(filename: "inner-subnest-enumeration.ll", directory: "")
+!2 = !{}
+!3 = !{i32 2, !"Debug Info Version", i32 3}
+!4 = !DISubroutineType(types: !2)
+!5 = distinct !DISubprogram(name: "deep_spine_wide_siblings", scope: !1, file: !1, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !2)
+!6 = !DILocation(line: 10, column: 1, scope: !5)
+!7 = !DILocation(line: 20, column: 1, scope: !5)
+!8 = !DILocation(line: 30, column: 1, scope: !5)
+!9 = !DILocation(line: 40, column: 1, scope: !5)
+!10 = !DILocation(line: 50, column: 1, scope: !5)
+!11 = !DILocation(line: 60, column: 1, scope: !5)
+!12 = !DILocation(line: 70, column: 1, scope: !5)
+!13 = !DILocation(line: 80, column: 1, scope: !5)
diff --git a/llvm/test/Transforms/LoopInterchange/inner-subnest-fallback-switch.ll b/llvm/test/Transforms/LoopInterchange/inner-subnest-fallback-switch.ll
new file mode 100644
index 0000000000000..9e46a2d733d12
--- /dev/null
+++ b/llvm/test/Transforms/LoopInterchange/inner-subnest-fallback-switch.ll
@@ -0,0 +1,84 @@
+; REQUIRES: asserts
+;
+; RUN: opt < %s -passes=loop-interchange -debug-only=loop-interchange \
+; RUN: -print-after-all -disable-output 2>&1 | \
+; RUN: FileCheck %s --check-prefix=ENABLED
+; RUN: opt < %s -passes=loop-interchange -debug-only=loop-interchange \
+; RUN: -loop-interchange-enable-inner-subnest-fallback=false \
+; RUN: -print-after-all -disable-output 2>&1 | \
+; RUN: FileCheck %s --check-prefix=DISABLED \
+; RUN: --implicit-check-not='Considering inner-subnest fallback'
+;
+; ENABLED: Considering inner-subnest fallback for loop nest
+; ENABLED: IR Dump After LoopInterchangePass
+;
+; DISABLED-NOT: Considering inner-subnest fallback for loop nest
+; DISABLED: IR Dump After LoopInterchangePass
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+
+ at A = global [4 x [8 x [8 x double]]] zeroinitializer
+ at B = global [4 x [8 x [8 x double]]] zeroinitializer
+
+define void @fallback_switch() {
+entry:
+ br label %root.header
+
+root.header:
+ %k = phi i64 [ 0, %entry ], [ %k.next, %root.latch ]
+ br label %a.i.header
+
+a.i.header:
+ %ai = phi i64 [ 0, %root.header ], [ %ai.next, %a.i.latch ]
+ br label %a.j.body
+
+a.j.body:
+ %aj = phi i64 [ 1, %a.i.header ], [ %aj.next, %a.j.body ]
+ %aj.prev = sub i64 %aj, 1
+ %a.load.ptr = getelementptr inbounds [4 x [8 x [8 x double]]],
+ ptr @A, i64 0, i64 %k, i64 %aj, i64 %ai
+ %a.value = load double, ptr %a.load.ptr, align 8
+ %a.next = fadd double %a.value, 1.000000e+00
+ %a.store.ptr = getelementptr inbounds [4 x [8 x [8 x double]]],
+ ptr @A, i64 0, i64 %k, i64 %aj.prev, i64 %ai
+ store double %a.next, ptr %a.store.ptr, align 8
+ %aj.next = add i64 %aj, 1
+ %aj.done = icmp eq i64 %aj.next, 8
+ br i1 %aj.done, label %a.i.latch, label %a.j.body
+
+a.i.latch:
+ %ai.next = add i64 %ai, 1
+ %ai.done = icmp eq i64 %ai.next, 7
+ br i1 %ai.done, label %b.i.header, label %a.i.header
+
+b.i.header:
+ %bi = phi i64 [ 0, %a.i.latch ], [ %bi.next, %b.i.latch ]
+ br label %b.j.body
+
+b.j.body:
+ %bj = phi i64 [ 1, %b.i.header ], [ %bj.next, %b.j.body ]
+ %bj.prev = sub i64 %bj, 1
+ %b.load.ptr = getelementptr inbounds [4 x [8 x [8 x double]]],
+ ptr @B, i64 0, i64 %k, i64 %bj, i64 %bi
+ %b.value = load double, ptr %b.load.ptr, align 8
+ %b.next = fadd double %b.value, 1.000000e+00
+ %b.store.ptr = getelementptr inbounds [4 x [8 x [8 x double]]],
+ ptr @B, i64 0, i64 %k, i64 %bj.prev, i64 %bi
+ store double %b.next, ptr %b.store.ptr, align 8
+ %bj.next = add i64 %bj, 1
+ %bj.done = icmp eq i64 %bj.next, 8
+ br i1 %bj.done, label %b.i.latch, label %b.j.body
+
+b.i.latch:
+ %bi.next = add i64 %bi, 1
+ %bi.done = icmp eq i64 %bi.next, 7
+ br i1 %bi.done, label %root.latch, label %b.i.header
+
+root.latch:
+ %k.next = add i64 %k, 1
+ %k.done = icmp eq i64 %k.next, 4
+ br i1 %k.done, label %exit, label %root.header
+
+exit:
+ ret void
+}
diff --git a/llvm/test/Transforms/LoopInterchange/large-nested-6d.ll b/llvm/test/Transforms/LoopInterchange/large-nested-6d.ll
index 590c21fd5a1be..ee6fda2496466 100644
--- a/llvm/test/Transforms/LoopInterchange/large-nested-6d.ll
+++ b/llvm/test/Transforms/LoopInterchange/large-nested-6d.ll
@@ -1,5 +1,8 @@
; RUN: opt < %s -passes=loop-interchange -cache-line-size=64 -pass-remarks='loop-interchange' -pass-remarks-missed='loop-interchange' -pass-remarks-output=%t -disable-output -S
-; RUN: FileCheck --input-file=%t %s
+; The whole nest is conservatively left unchanged: every eligible inner pair is
+; rejected (unknown surrounding context or dependences), so no `Interchanged`
+; (`!Passed`) record may appear anywhere in the remark stream.
+; RUN: FileCheck --input-file=%t %s --implicit-check-not=Interchanged --implicit-check-not='!Passed'
target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128-Fn32"
@@ -43,10 +46,10 @@ target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i8:8:32-i16:16:32-i6
; There are a few issues that prevent loop-interchange to perform its
; transformation on this test case:
;
-; 1. LoopNest checks: the first check that is perform is whether loop 'L.header'
-; and 'M.header' are perfectly nested, which they are not. It needs to be
-; investigate why the whole loop nest rooted under L is rejected as a
-; candidate.
+; 1. Candidate formation: the full breadth-first nest is non-linear. The
+; inner-subnest fallback now considers direct inner pairs, but the relevant
+; candidates are conservatively rejected because their surrounding
+; dependence context is unknown.
;
; 2. DependenceAnalysis: it finds this dependency:
;
@@ -55,16 +58,27 @@ target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i8:8:32-i16:16:32-i6
; Dst: store double %46, ptr %48, align 8
;
;
+; CHECK-NOT: --- !Analysis
; CHECK: --- !Missed
; CHECK-NEXT: Pass: loop-interchange
-; CHECK-NEXT: Name: UnsupportedLoopNestDepth
+; CHECK-NEXT: Name: Dependence
; CHECK-NEXT: Function: test
; CHECK-NEXT: Args:
-; CHECK-NEXT: - String: 'Unsupported depth of loop nest, the supported range is ['
-; CHECK-NEXT: - String: '2'
-; CHECK-NEXT: - String: ', '
-; CHECK-NEXT: - String: '10'
-; CHECK-NEXT: - String: "].\n"
+; CHECK-NEXT: - String: All loops have dependencies in all directions.
+; CHECK-NEXT: ...
+; CHECK-NEXT: --- !Missed
+; CHECK-NEXT: Pass: loop-interchange
+; CHECK-NEXT: Name: FallbackUnknownContext
+; CHECK-NEXT: Function: test
+; CHECK-NEXT: Args:
+; CHECK-NEXT: - String: 'Cannot interchange inner subnest: the surrounding dependence context is unknown or unsafe.'
+; CHECK-NEXT: ...
+; CHECK-NEXT: --- !Missed
+; CHECK-NEXT: Pass: loop-interchange
+; CHECK-NEXT: Name: FallbackUnknownContext
+; CHECK-NEXT: Function: test
+; CHECK-NEXT: Args:
+; CHECK-NEXT: - String: 'Cannot interchange inner subnest: the surrounding dependence context is unknown or unsafe.'
; CHECK-NEXT: ...
; CHECK-NEXT: --- !Analysis
; CHECK-NEXT: Pass: loop-interchange
More information about the llvm-commits
mailing list