[llvm] Macro fusion early exist on any cluster (PR #191710)
Tomer Shafir via llvm-commits
llvm-commits at lists.llvm.org
Sun Apr 12 07:56:27 PDT 2026
https://github.com/tomershafir updated https://github.com/llvm/llvm-project/pull/191710
>From d50b844eaad959f6411ffb7f5fe7b347e5dc2a62 Mon Sep 17 00:00:00 2001
From: tomershafir <tomer.shafir8 at gmail.com>
Date: Sat, 11 Apr 2026 21:17:13 +0300
Subject: [PATCH 1/2] [MISched] Extract `isClustered()` method on SUnit (NFC)
This patch encapsulates the check for wether a `SUnit` is clustered, rather than letting it scatter across call sites. Currently there is only a single user, but more users can show up, and I think it provides a cleaner API even for that single user.
---
llvm/include/llvm/CodeGen/ScheduleDAG.h | 2 ++
llvm/lib/CodeGen/ScheduleDAG.cpp | 2 +-
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/llvm/include/llvm/CodeGen/ScheduleDAG.h b/llvm/include/llvm/CodeGen/ScheduleDAG.h
index aee1514581485..9ec7e417054d9 100644
--- a/llvm/include/llvm/CodeGen/ScheduleDAG.h
+++ b/llvm/include/llvm/CodeGen/ScheduleDAG.h
@@ -481,6 +481,8 @@ class TargetRegisterInfo;
/// edge occurs first.
LLVM_ABI void biasCriticalPath();
+ bool isClustered() const { return ParentClusterIdx != InvalidClusterId; }
+
LLVM_ABI void dumpAttributes() const;
private:
diff --git a/llvm/lib/CodeGen/ScheduleDAG.cpp b/llvm/lib/CodeGen/ScheduleDAG.cpp
index e630b80e33ab4..7008d93dd8aca 100644
--- a/llvm/lib/CodeGen/ScheduleDAG.cpp
+++ b/llvm/lib/CodeGen/ScheduleDAG.cpp
@@ -365,7 +365,7 @@ LLVM_DUMP_METHOD void ScheduleDAG::dumpNodeName(const SUnit &SU) const {
LLVM_DUMP_METHOD void ScheduleDAG::dumpNodeAll(const SUnit &SU) const {
dumpNode(SU);
SU.dumpAttributes();
- if (SU.ParentClusterIdx != InvalidClusterId)
+ if (SU.isClustered())
dbgs() << " Parent Cluster Index: " << SU.ParentClusterIdx << '\n';
if (SU.Preds.size() > 0) {
>From b07a132d4ed581ffa274ee7c4e0c8f154ed0e7fe Mon Sep 17 00:00:00 2001
From: tomershafir <tomer.shafir8 at gmail.com>
Date: Sun, 12 Apr 2026 14:13:26 +0300
Subject: [PATCH 2/2] [MacroFusion] Early return when insts already clustered
This patch adds an early return to `fuseInstructionPair()` when macro fused instructions are already clustered, either by an earlier fusion or another clustering like ld/st clustering, and removing the assert.
The assert is generally wrong - there are cases where an earlier ld/st clustering (before macro fusion) reached the assert because it sets `ParentClusterIdx` and fails.
Currently, clustering support limits instruction membership to at most a single cluster through `ParentClusterIdx`. Thus, previously asserts-on builds would effectively reject fusions of already clustered instructions, but asserts-off builds could override earlier fusions/other-clustering, inconsistently with each other.
This patch applies a global FIFO clustering order from MacroFusion's perspective for all build configs:
1. Early fusions win - I think this is an appropriate generic heuristic, to try greedily maximize fusion oppurtunities. At least until we have a way to compare fusion candidates - of course we can miss better schedules in some cases without a broader search.
2. Early clusterings win - currently, the only other clusterings are for ld/st, and they performed before macro fusion in the pipeline. I think they should be generally preferred over fusions, because they not only save dispatch and execution slots, but they also should decrease code size. If a new clustering algorithm is added, it would have to inspect ordering and be placed correctly.
For the reference, Ld/st clustering applies a similar heuristic, prioritizing early clusters. The difference is that loads and stores are mutually exclusive, compared to fusions and ld/st clustering.
---
llvm/lib/CodeGen/MacroFusion.cpp | 28 +++----
.../AArch64/macro-fusion-cluster-conflict.mir | 75 +++++++++++++++++++
2 files changed, 87 insertions(+), 16 deletions(-)
create mode 100644 llvm/test/CodeGen/AArch64/macro-fusion-cluster-conflict.mir
diff --git a/llvm/lib/CodeGen/MacroFusion.cpp b/llvm/lib/CodeGen/MacroFusion.cpp
index 1db53017e6cef..55dadea676dbb 100644
--- a/llvm/lib/CodeGen/MacroFusion.cpp
+++ b/llvm/lib/CodeGen/MacroFusion.cpp
@@ -25,6 +25,9 @@
#define DEBUG_TYPE "machine-scheduler"
STATISTIC(NumFused, "Number of instr pairs fused");
+STATISTIC(NumFusionConflicts,
+ "Number of conflicts between a fusion pair and an already existing "
+ "cluster (either fusion or non-fusion)");
using namespace llvm;
@@ -52,22 +55,15 @@ bool llvm::hasLessThanNumFused(const SUnit &SU, unsigned FuseLimit) {
bool llvm::fuseInstructionPair(ScheduleDAGInstrs &DAG, SUnit &FirstSU,
SUnit &SecondSU) {
- // Check that neither instr is already paired with another along the edge
- // between them.
- for (SDep &SI : FirstSU.Succs)
- if (SI.isCluster())
- return false;
-
- for (SDep &SI : SecondSU.Preds)
- if (SI.isCluster())
- return false;
-
- assert(FirstSU.ParentClusterIdx == InvalidClusterId &&
- SecondSU.ParentClusterIdx == InvalidClusterId);
-
- // Though the reachability checks above could be made more generic,
- // perhaps as part of ScheduleDAGInstrs::addEdge(), since such edges are valid,
- // the extra computation cost makes it less interesting in general cases.
+ // Check that neither instr is already associated with a cluster (either
+ // fusion or non-fusion)
+ if (FirstSU.isClustered() || SecondSU.isClustered()) {
+ ++NumFusionConflicts;
+ LLVM_DEBUG(dbgs() << "Fusion conflict: cannot fuse SU(" << FirstSU.NodeNum
+ << ") and SU(" << SecondSU.NodeNum
+ << ") - already clustered\n");
+ return false;
+ }
// Create a single weak edge between the adjacent instrs. The only effect is
// to cause bottom-up scheduling to heavily prioritize the clustered instrs.
diff --git a/llvm/test/CodeGen/AArch64/macro-fusion-cluster-conflict.mir b/llvm/test/CodeGen/AArch64/macro-fusion-cluster-conflict.mir
new file mode 100644
index 0000000000000..5dd3a3c9c3e28
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/macro-fusion-cluster-conflict.mir
@@ -0,0 +1,75 @@
+# REQUIRES: aarch64-registered-target
+# REQUIRES: asserts
+
+# RUN: llc -o /dev/null %s -mtriple=aarch64 -mattr=+fuse-address \
+# RUN: -run-pass machine-scheduler -debug-only=machine-scheduler 2>&1 \
+# RUN: | FileCheck %s
+# RUN: llc -o /dev/null %s -mtriple=aarch64-apple-macos -mcpu=apple-m5 \
+# RUN: -run-pass machine-scheduler -debug-only=machine-scheduler 2>&1 \
+# RUN: | FileCheck %s
+
+# Test macro fusion conflicts with other macro fusions and with ld/st clustering.
+
+# Fusion-fusion conflict: ADRP fans out to two loads at non-adjacent offsets
+# (so they are NOT clustered by LoadClusterDAGMutation). Both ADRP→LDR pairs
+# are valid address fusion pairs. SU(0)→SU(1) fuses first; when SU(0)→SU(2)
+# is attempted, SU(0) is already clustered.
+---
+name: fusion-fusion-conflict
+tracksRegLiveness: true
+body: |
+ bb.0:
+ ; CHECK-LABEL: fusion-fusion-conflict
+ ; CHECK: Macro fuse: SU(0) - SU(1) / ADRP - LDRXui
+ ; CHECK: Fusion conflict: cannot fuse SU(0) and SU(2) - already clustered
+ ; CHECK: SU(0): $x0 = ADRP 0
+ ; CHECK: SU(1): $x1 = LDRXui $x0, 5
+ ; CHECK: SU(2): $x2 = LDRXui $x0, 10
+ $x0 = ADRP 0
+ $x1 = LDRXui $x0, 5 :: (load (s64))
+ $x2 = LDRXui $x0, 10 :: (load (s64))
+ RET_ReallyLR
+...
+
+# Store-fusion conflict: two adjacent stores get clustered by
+# StoreClusterDAGMutation (runs before AArch64 macro fusion). Then address
+# fusion (ADRP + STRXui) is attempted but the store SUnit is already
+# clustered.
+---
+name: store-fusion-conflict
+tracksRegLiveness: true
+body: |
+ bb.0:
+ liveins: $x0
+
+ ; CHECK-LABEL: store-fusion-conflict
+ ; CHECK: Cluster ld/st SU(1) - SU(2)
+ ; CHECK: Fusion conflict: cannot fuse SU(0) and SU(1) - already clustered
+ ; CHECK: SU(0): $x1 = ADRP 0
+ ; CHECK: SU(1): STRXui $x0, $x1, 0 :: (store (s64))
+ ; CHECK: SU(2): STRXui $x0, $x1, 1 :: (store (s64))
+ $x1 = ADRP 0
+ STRXui $x0, $x1, 0 :: (store (s64))
+ STRXui $x0, $x1, 1 :: (store (s64))
+ RET_ReallyLR
+...
+
+# Load-fusion conflict: two adjacent loads get clustered by
+# LoadClusterDAGMutation. Then address fusion (ADRP + LDRXui) is attempted
+# but the load SUnit is already clustered.
+---
+name: load-fusion-conflict
+tracksRegLiveness: true
+body: |
+ bb.0:
+ ; CHECK-LABEL: load-fusion-conflict
+ ; CHECK: Cluster ld/st SU(1) - SU(2)
+ ; CHECK: Fusion conflict: cannot fuse SU(0) and SU(1) - already clustered
+ ; CHECK: SU(0): $x0 = ADRP 0
+ ; CHECK: SU(1): $x1 = LDRXui $x0, 0
+ ; CHECK: SU(2): $x2 = LDRXui $x0, 1
+ $x0 = ADRP 0
+ $x1 = LDRXui $x0, 0 :: (load (s64))
+ $x2 = LDRXui $x0, 1 :: (load (s64))
+ RET_ReallyLR
+...
More information about the llvm-commits
mailing list