[llvm] [LoopInterchange] Do not re-nest freeze instructions (PR #213309)

via llvm-commits llvm-commits at lists.llvm.org
Fri Jul 31 09:46:16 PDT 2026


https://github.com/MattPD created https://github.com/llvm/llvm-project/pull/213309

## Summary

Reject loop interchange when it would change the dynamic sampling of a
`freeze` or duplicate one as part of inner-latch reconstruction.

The check is deliberately narrower than rejecting every `freeze` in the nest:

- original outer-header/latch and inner-preheader/exit bodies change execution
  multiplicity and reject;
- the inner-latch condition and induction operand closure is duplicated and
  rejects;
- the original outer-preheader body remains outside the new outer loop after
  preheader-body swapping and stays legal;
- an ordinary innermost body retains the trip-count product and stays legal.

## Rationale

`freeze` chooses one stable value for a dynamic execution. Re-nesting it can
resample poison, while cloning it can break correlation between the original
and duplicate. Both are semantic changes even though `freeze` has no side
effects.

The legality check mirrors the blocks whose execution count changes and the
operand closure cloned by the transform. It does not alter profitability or
ordinary interchange behavior for nests without an affected `freeze`.

## Tests

- Add a profitable control and a live outer-preheader `freeze` positive.
- Reject a live outer-header `freeze` with an `UnsafeInst` remark.
- Require byte-identical IR when that unsafe case is declined.
- Preserve the existing innermost-body-`freeze` profitability decision: it is
  still declined as `InterchangeNotProfitable`, not the new freeze rejection.

The complete submission stack passes `check-llvm` (55,651), `check-flang`
(4,672), Scalar unit tests (15/15), and Utils unit tests (186/186), with no
unexpected failures.


>From fb92482f66360e5677f6148edd57ebfc347e1fc1 Mon Sep 17 00:00:00 2001
From: "Matt P. Dziubinski" <matt-p.dziubinski at hpe.com>
Date: Fri, 31 Jul 2026 06:44:00 -0500
Subject: [PATCH] [LoopInterchange] Do not re-nest freeze instructions

Reject freezes only when interchange changes their dynamic execution count or duplicates them. This covers re-nested control blocks and the inner-latch clone closure while preserving safe inner-body and outer-preheader freezes.

Add profitable positive controls and a byte-identity rejection oracle.

Co-authored-by: Copilot <223556219+Copilot at users.noreply.github.com>
---
 .../lib/Transforms/Scalar/LoopInterchange.cpp |  77 ++++++++++
 .../test/Transforms/LoopInterchange/freeze.ll | 135 ++++++++++++++++++
 .../profitability-vectorization-heuristic.ll  |   9 ++
 3 files changed, 221 insertions(+)
 create mode 100644 llvm/test/Transforms/LoopInterchange/freeze.ll

diff --git a/llvm/lib/Transforms/Scalar/LoopInterchange.cpp b/llvm/lib/Transforms/Scalar/LoopInterchange.cpp
index c79bf04df656a..28a22e72ae1aa 100644
--- a/llvm/lib/Transforms/Scalar/LoopInterchange.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopInterchange.cpp
@@ -506,6 +506,9 @@ class LoopInterchangeLegality {
 private:
   bool tightlyNested(Loop *Outer, Loop *Inner);
   bool containsUnsafeInstructions(BasicBlock *BB, Instruction *Skip);
+  FreezeInst *findFreezeInReNestedBlocks(Loop *OuterLoop,
+                                         Loop *InnerLoop) const;
+  FreezeInst *findFreezeInInnerLatchCloneSet(Loop *InnerLoop) const;
 
   /// Traverse all PHI nodes in the header of each loop in the loop nest
   /// starting from \p OuterLoop, and perform the following checks:
@@ -813,6 +816,65 @@ bool LoopInterchangeLegality::containsUnsafeInstructions(BasicBlock *BB,
   });
 }
 
+FreezeInst *
+LoopInterchangeLegality::findFreezeInReNestedBlocks(Loop *OuterLoop,
+                                                    Loop *InnerLoop) const {
+  // adjustLoopLinks swaps the preheader bodies after changing their loop
+  // roles, so the original outer-preheader body remains outside the new outer
+  // loop and retains its execution count.
+  BasicBlock *Blocks[] = {
+      OuterLoop->getHeader(),
+      OuterLoop->getLoopLatch(),
+      InnerLoop->getLoopPreheader(),
+      InnerLoop->getExitBlock(),
+  };
+  for (BasicBlock *BB : Blocks)
+    if (BB)
+      for (Instruction &I : *BB)
+        if (auto *Freeze = dyn_cast<FreezeInst>(&I))
+          return Freeze;
+  return nullptr;
+}
+
+FreezeInst *
+LoopInterchangeLegality::findFreezeInInnerLatchCloneSet(Loop *InnerLoop) const {
+  // Trip-count and induction legality currently reject most freezes reachable
+  // here. Keep this worklist aligned with the transform's clone set so future
+  // analyzable latch forms cannot silently resample one.
+  SmallSetVector<Instruction *, 8> Worklist;
+  auto IsDirectInnerLoopBlock = [InnerLoop](BasicBlock *BB) {
+    return InnerLoop->contains(BB) &&
+           none_of(InnerLoop->getSubLoops(),
+                   [BB](Loop *SubLoop) { return SubLoop->contains(BB); });
+  };
+  auto *LatchBranch =
+      dyn_cast<CondBrInst>(InnerLoop->getLoopLatch()->getTerminator());
+  if (LatchBranch)
+    if (auto *Condition = dyn_cast<Instruction>(LatchBranch->getCondition()))
+      Worklist.insert(Condition);
+
+  for (PHINode *Induction : InnerLoopInductions) {
+    auto *Incoming = dyn_cast<Instruction>(
+        Induction->getIncomingValueForBlock(InnerLoop->getLoopLatch()));
+    if (Incoming && !is_contained(InnerLoopInductions, Incoming))
+      Worklist.insert(Incoming);
+  }
+
+  for (unsigned I = 0; I < Worklist.size(); ++I) {
+    Instruction *Current = Worklist[I];
+    if (auto *Freeze = dyn_cast<FreezeInst>(Current))
+      return Freeze;
+    for (Value *Operand : Current->operands()) {
+      auto *OperandI = dyn_cast<Instruction>(Operand);
+      if (!OperandI || !IsDirectInnerLoopBlock(OperandI->getParent()) ||
+          is_contained(InnerLoopInductions, OperandI))
+        continue;
+      Worklist.insert(OperandI);
+    }
+  }
+  return nullptr;
+}
+
 bool LoopInterchangeLegality::tightlyNested(Loop *OuterLoop, Loop *InnerLoop) {
   BasicBlock *OuterLoopHeader = OuterLoop->getHeader();
   BasicBlock *InnerLoopPreHeader = InnerLoop->getLoopPreheader();
@@ -1583,6 +1645,21 @@ bool LoopInterchangeLegality::canInterchangeLoops(unsigned InnerLoopId,
     return false;
   }
 
+  FreezeInst *Freeze = findFreezeInReNestedBlocks(OuterLoop, InnerLoop);
+  if (!Freeze)
+    Freeze = findFreezeInInnerLatchCloneSet(InnerLoop);
+  if (Freeze) {
+    LLVM_DEBUG(dbgs() << "Interchange would re-nest or duplicate freeze\n");
+    ORE->emit([&]() {
+      return OptimizationRemarkMissed(DEBUG_TYPE, "UnsafeInst",
+                                      Freeze->getDebugLoc(),
+                                      Freeze->getParent())
+             << "Cannot interchange loops because re-nesting or duplicating "
+                "freeze may change its sampling behavior.";
+    });
+    return false;
+  }
+
   // TODO: The loops could not be interchanged due to current limitations in the
   // transform module.
   if (currentLimitations()) {
diff --git a/llvm/test/Transforms/LoopInterchange/freeze.ll b/llvm/test/Transforms/LoopInterchange/freeze.ll
new file mode 100644
index 0000000000000..671e95472f52a
--- /dev/null
+++ b/llvm/test/Transforms/LoopInterchange/freeze.ll
@@ -0,0 +1,135 @@
+; A freeze whose execution count changes under interchange can resample poison
+; and break the correlation shared by its uses. Reject that nest, but retain
+; interchange for an otherwise-identical profitable control.
+;
+; RUN: opt < %s -passes='loop(loop-interchange),print<loops>' \
+; RUN:     -cache-line-size=64 -disable-output 2>&1 \
+; RUN:     | FileCheck %s --check-prefix=LOOPS
+; RUN: opt < %s -passes=loop-interchange -cache-line-size=64 \
+; RUN:     -pass-remarks-output=%t.yaml -disable-output
+; RUN: FileCheck %s --check-prefix=REMARK --input-file=%t.yaml
+; RUN: llvm-extract -S -func=outer_header_freeze %s -o %t.freeze
+; RUN: opt -S -passes=no-op-loopnest %t.freeze -o %t.noop
+; RUN: opt -S -passes=loop-interchange -cache-line-size=64 \
+; RUN:     %t.freeze -o %t.out
+; RUN: diff -u %t.noop %t.out
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+
+define void @profitable_control(ptr noalias %A, ptr noalias %R) {
+entry:
+  br label %outer.header
+
+outer.header:
+  %i = phi i64 [ 0, %entry ], [ %i.next, %outer.latch ]
+  %sum.i = phi double [ 0.000000e+00, %entry ], [ %sum.next, %outer.latch ]
+  br label %inner.header
+
+inner.header:
+  %j = phi i64 [ 0, %outer.header ], [ %j.next, %inner.header ]
+  %sum.j = phi double [ %sum.i, %outer.header ], [ %sum.next.j, %inner.header ]
+  %idx = getelementptr inbounds [4 x double], ptr %A, i64 %j, i64 %i
+  %value = load double, ptr %idx, align 8
+  %sum.next.j = fadd reassoc double %sum.j, %value
+  %j.next = add i64 %j, 1
+  %j.ec = icmp eq i64 %j.next, 4
+  br i1 %j.ec, label %outer.latch, label %inner.header
+
+outer.latch:
+  %sum.next = phi double [ %sum.next.j, %inner.header ]
+  %i.next = add i64 %i, 1
+  %i.ec = icmp eq i64 %i.next, 4
+  br i1 %i.ec, label %exit, label %outer.header
+
+exit:
+  %sum.result = phi double [ %sum.next, %outer.latch ]
+  store double %sum.result, ptr %R, align 8
+  ret void
+}
+
+; LOOPS-LABEL: Loop info for function 'profitable_control':
+; LOOPS:         Loop at depth 1 containing: %inner.header<header>
+; LOOPS-NEXT:      Loop at depth 2 containing: %outer.header<header>
+; REMARK:      --- !Passed
+; REMARK:      Name:            Interchanged
+; REMARK-NEXT: Function:        profitable_control
+
+define void @outer_preheader_freeze(ptr noalias %A, ptr noalias %R) {
+entry:
+  %choice = freeze i1 poison
+  br label %outer.header
+
+outer.header:
+  %i = phi i64 [ 0, %entry ], [ %i.next, %outer.latch ]
+  %sum.i = phi double [ 0.000000e+00, %entry ], [ %sum.next, %outer.latch ]
+  br label %inner.header
+
+inner.header:
+  %j = phi i64 [ 0, %outer.header ], [ %j.next, %inner.header ]
+  %sum.j = phi double [ %sum.i, %outer.header ], [ %sum.next.j, %inner.header ]
+  %idx = getelementptr inbounds [4 x double], ptr %A, i64 %j, i64 %i
+  %value = load double, ptr %idx, align 8
+  %selected = select i1 %choice, double %value, double 0.000000e+00
+  %sum.next.j = fadd reassoc double %sum.j, %selected
+  %j.next = add i64 %j, 1
+  %j.ec = icmp eq i64 %j.next, 4
+  br i1 %j.ec, label %outer.latch, label %inner.header
+
+outer.latch:
+  %sum.next = phi double [ %sum.next.j, %inner.header ]
+  %i.next = add i64 %i, 1
+  %i.ec = icmp eq i64 %i.next, 4
+  br i1 %i.ec, label %exit, label %outer.header
+
+exit:
+  %sum.result = phi double [ %sum.next, %outer.latch ]
+  store double %sum.result, ptr %R, align 8
+  ret void
+}
+
+; LOOPS-LABEL: Loop info for function 'outer_preheader_freeze':
+; LOOPS:         Loop at depth 1 containing: %inner.header<header>
+; LOOPS-NEXT:      Loop at depth 2 containing: %outer.header<header>
+; REMARK:      --- !Passed
+; REMARK:      Name:            Interchanged
+; REMARK-NEXT: Function:        outer_preheader_freeze
+
+define void @outer_header_freeze(ptr noalias %A, ptr noalias %R) {
+entry:
+  br label %outer.header
+
+outer.header:
+  %i = phi i64 [ 0, %entry ], [ %i.next, %outer.latch ]
+  %sum.i = phi double [ 0.000000e+00, %entry ], [ %sum.next, %outer.latch ]
+  %choice = freeze i1 poison
+  br label %inner.header
+
+inner.header:
+  %j = phi i64 [ 0, %outer.header ], [ %j.next, %inner.header ]
+  %sum.j = phi double [ %sum.i, %outer.header ], [ %sum.next.j, %inner.header ]
+  %idx = getelementptr inbounds [4 x double], ptr %A, i64 %j, i64 %i
+  %value = load double, ptr %idx, align 8
+  %selected = select i1 %choice, double %value, double 0.000000e+00
+  %sum.next.j = fadd reassoc double %sum.j, %selected
+  %j.next = add i64 %j, 1
+  %j.ec = icmp eq i64 %j.next, 4
+  br i1 %j.ec, label %outer.latch, label %inner.header
+
+outer.latch:
+  %sum.next = phi double [ %sum.next.j, %inner.header ]
+  %i.next = add i64 %i, 1
+  %i.ec = icmp eq i64 %i.next, 4
+  br i1 %i.ec, label %exit, label %outer.header
+
+exit:
+  %sum.result = phi double [ %sum.next, %outer.latch ]
+  store double %sum.result, ptr %R, align 8
+  ret void
+}
+
+; LOOPS-LABEL: Loop info for function 'outer_header_freeze':
+; LOOPS:         Loop at depth 1 containing: %outer.header<header>
+; LOOPS-NEXT:      Loop at depth 2 containing: %inner.header<header>
+; REMARK:      --- !Missed
+; REMARK:      Name:            UnsafeInst
+; REMARK-NEXT: Function:        outer_header_freeze
diff --git a/llvm/test/Transforms/LoopInterchange/profitability-vectorization-heuristic.ll b/llvm/test/Transforms/LoopInterchange/profitability-vectorization-heuristic.ll
index 342a9e3fb49f6..e2b8ce2f4ee53 100644
--- a/llvm/test/Transforms/LoopInterchange/profitability-vectorization-heuristic.ll
+++ b/llvm/test/Transforms/LoopInterchange/profitability-vectorization-heuristic.ll
@@ -1,5 +1,11 @@
 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
 ; RUN: opt < %s -passes=loop-interchange -cache-line-size=64 -loop-interchange-profitabilities=vectorize -S | FileCheck %s
+; RUN: llvm-extract -S -func=multiple_BBs_in_loop %s -o %t.multibb
+; RUN: opt < %t.multibb -passes=loop-interchange -cache-line-size=64 \
+; RUN:     -loop-interchange-profitabilities=vectorize \
+; RUN:     -pass-remarks-output=%t.multibb.yaml -disable-output
+; RUN: FileCheck %s --check-prefix=MULTIBB --input-file=%t.multibb.yaml \
+; RUN:     --implicit-check-not=UnsafeInst
 
 @A = dso_local global [256 x [256 x float]] zeroinitializer
 @B = dso_local global [256 x [256 x float]] zeroinitializer
@@ -268,6 +274,9 @@ exit:
 ; }
 
 define void @multiple_BBs_in_loop() {
+; MULTIBB:      --- !Missed
+; MULTIBB:      Name:            InterchangeNotProfitable
+; MULTIBB-NEXT: Function:        multiple_BBs_in_loop
 ; CHECK-LABEL: define void @multiple_BBs_in_loop() {
 ; CHECK-NEXT:  [[ENTRY:.*]]:
 ; CHECK-NEXT:    br label %[[FOR_I_HEADER:.*]]



More information about the llvm-commits mailing list