[llvm] [LoopIdiomVectorize] Don't add the match-index block to the parent loop when it exits (PR #225576)
Kunal Pathak via llvm-commits
llvm-commits at lists.llvm.org
Thu Sep 24 12:06:43 PDT 2026
https://github.com/kunalspathak updated https://github.com/llvm/llvm-project/pull/225576
>From a26543b4dd4ea5dfb3a6a9c04eace860144edd32 Mon Sep 17 00:00:00 2001
From: Kunal Pathak <kupathak at meta.com>
Date: Tue, 22 Sep 2026 19:40:13 -0700
Subject: [PATCH 1/3] LoopIdiomVectorize] Don't add the match-index block to
the parent loop when it exits
`expandFindFirstByte` unconditionally adds BB4, the block that computes the index of the match, to the parent loop. BB4 branches only to ExitSucc. When the
`find_first_of` idiom is nested inside another loop, and a match exits that enclosing loop, BB4 always leaves the parent loop and so is not part of it. `LoopInfo`
is then inconsistent, and `LoopBase::verifyLoop()` fails with `"Loop block has no in-loop successors!"`.
A release build compiles that check out. It instead crashes later, in a pass that consumes the stale analysis. IndVarSimplify is the one seen in practice.
Only add BB4 to the parent loop when the parent loop contains ExitSucc. The sibling `expandFindMismatch`` is unaffected, because its success path lands in a
block split out of the preheader, which is inside the parent loop already.
This is a regression from #101976. It reproduces from 21.1.0 through main.
It needs `+sve2` in practice, because the expansion is gated on the cost of `llvm.experimental.vector.match``.
Fixes: https://github.com/llvm/llvm-project/issues/225571
---
.../Vectorize/LoopIdiomVectorize.cpp | 6 +-
.../AArch64/find-first-byte-nested.ll | 253 ++++++++++++++++++
2 files changed, 258 insertions(+), 1 deletion(-)
create mode 100644 llvm/test/Transforms/LoopIdiom/AArch64/find-first-byte-nested.ll
diff --git a/llvm/lib/Transforms/Vectorize/LoopIdiomVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopIdiomVectorize.cpp
index 6f59921d9a11e8..2854e50a2613d8 100644
--- a/llvm/lib/Transforms/Vectorize/LoopIdiomVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopIdiomVectorize.cpp
@@ -1253,7 +1253,11 @@ Value *LoopIdiomVectorize::expandFindFirstByte(
if (auto ParentLoop = CurLoop->getParentLoop()) {
ParentLoop->addBasicBlockToLoop(BB0, *LI);
ParentLoop->addChildLoop(OuterLoop);
- ParentLoop->addBasicBlockToLoop(BB4, *LI);
+ // BB4 branches only to ExitSucc, so it belongs to the parent loop only when
+ // that exit is itself inside the parent loop. Otherwise BB4 always leaves,
+ // which would leave a parent block with no in-loop successor.
+ if (ParentLoop->contains(ExitSucc))
+ ParentLoop->addBasicBlockToLoop(BB4, *LI);
} else {
LI->addTopLevelLoop(OuterLoop);
}
diff --git a/llvm/test/Transforms/LoopIdiom/AArch64/find-first-byte-nested.ll b/llvm/test/Transforms/LoopIdiom/AArch64/find-first-byte-nested.ll
new file mode 100644
index 00000000000000..ff4febad22cef3
--- /dev/null
+++ b/llvm/test/Transforms/LoopIdiom/AArch64/find-first-byte-nested.ll
@@ -0,0 +1,253 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt -mtriple=aarch64 -mattr=+sve -passes='loop(loop-idiom-vectorize)' -verify-loop-info -verify-dom-info -S < %s | FileCheck %s
+
+; The find_first_of idiom nested inside an enclosing loop. The expansion adds
+; the block that computes the match index to the enclosing loop, but that block
+; branches only to the successful exit. When that exit leaves the enclosing
+; loop, the block always leaves it too and must not be added to it.
+
+define void @find_first_of_i8_nested_exit_outside(ptr %search_end, ptr %needle_end) #0 {
+; CHECK-LABEL: define void @find_first_of_i8_nested_exit_outside(
+; CHECK-SAME: ptr [[SEARCH_END:%.*]], ptr [[NEEDLE_END:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT: [[ENTRY:.*:]]
+; CHECK-NEXT: br label %[[OUTER_HEADER:.*]]
+; CHECK: [[OUTER_LATCH:.*]]:
+; CHECK-NEXT: br label %[[OUTER_HEADER]]
+; CHECK: [[OUTER_HEADER]]:
+; CHECK-NEXT: br label %[[MEM_CHECK:.*]]
+; CHECK: [[MEM_CHECK]]:
+; CHECK-NEXT: [[SEARCH_END_INT:%.*]] = ptrtoint ptr [[SEARCH_END]] to i64
+; CHECK-NEXT: [[TMP0:%.*]] = ptrtoaddr ptr [[SEARCH_END]] to i64
+; CHECK-NEXT: [[SEARCH_TRIP_COUNT:%.*]] = sub i64 [[TMP0]], 0
+; CHECK-NEXT: [[NEEDLE_END_INT:%.*]] = ptrtoint ptr [[NEEDLE_END]] to i64
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[NEEDLE_END]] to i64
+; CHECK-NEXT: [[NEEDLE_TRIP_COUNT:%.*]] = sub i64 [[TMP1]], 0
+; CHECK-NEXT: [[TMP2:%.*]] = call <vscale x 16 x i1> @llvm.get.active.lane.mask.nxv16i1.i64(i64 0, i64 16)
+; CHECK-NEXT: [[SEARCH_END_PAGE:%.*]] = lshr i64 [[SEARCH_END_INT]], 12
+; CHECK-NEXT: [[NEEDLE_END_PAGE:%.*]] = lshr i64 [[NEEDLE_END_INT]], 12
+; CHECK-NEXT: [[SEARCH_PAGE_CMP:%.*]] = icmp ne i64 0, [[SEARCH_END_PAGE]]
+; CHECK-NEXT: [[NEEDLE_PAGE_CMP:%.*]] = icmp ne i64 0, [[NEEDLE_END_PAGE]]
+; CHECK-NEXT: [[COMBINED_PAGE_CMP:%.*]] = or i1 [[SEARCH_PAGE_CMP]], [[NEEDLE_PAGE_CMP]]
+; CHECK-NEXT: br i1 [[COMBINED_PAGE_CMP]], label %[[SCALAR_PREHEADER:.*]], label %[[FIND_FIRST_VEC_HEADER:.*]], !prof [[PROF0:![0-9]+]]
+; CHECK: [[FIND_FIRST_VEC_HEADER]]:
+; CHECK-NEXT: [[SEARCH_IDX:%.*]] = phi i64 [ 0, %[[MEM_CHECK]] ], [ [[SEARCH_IDX_NEXT:%.*]], %[[SEARCH_CHECK_VEC:.*]] ]
+; CHECK-NEXT: [[SEARCH_PRED:%.*]] = call <vscale x 16 x i1> @llvm.get.active.lane.mask.nxv16i1.i64(i64 [[SEARCH_IDX]], i64 [[SEARCH_TRIP_COUNT]])
+; CHECK-NEXT: [[SEARCH_MASKED:%.*]] = and <vscale x 16 x i1> [[TMP2]], [[SEARCH_PRED]]
+; CHECK-NEXT: [[PSEARCH:%.*]] = getelementptr i8, ptr null, i64 [[SEARCH_IDX]]
+; CHECK-NEXT: [[SEARCH_LOAD_VEC:%.*]] = call <vscale x 16 x i8> @llvm.masked.load.nxv16i8.p0(ptr align 1 [[PSEARCH]], <vscale x 16 x i1> [[SEARCH_MASKED]], <vscale x 16 x i8> zeroinitializer)
+; CHECK-NEXT: br label %[[NEEDLE_CHECK_VEC:.*]]
+; CHECK: [[NEEDLE_CHECK_VEC]]:
+; CHECK-NEXT: [[NEEDLE_IDX:%.*]] = phi i64 [ 0, %[[FIND_FIRST_VEC_HEADER]] ], [ [[NEEDLE_IDX_NEXT:%.*]], %[[NEEDLE_CHECK_VEC]] ]
+; CHECK-NEXT: [[PMATCH:%.*]] = phi <vscale x 16 x i1> [ zeroinitializer, %[[FIND_FIRST_VEC_HEADER]] ], [ [[MATCH_ACCUMULATOR:%.*]], %[[NEEDLE_CHECK_VEC]] ]
+; CHECK-NEXT: [[NEEDLE_PRED:%.*]] = call <vscale x 16 x i1> @llvm.get.active.lane.mask.nxv16i1.i64(i64 [[NEEDLE_IDX]], i64 [[NEEDLE_TRIP_COUNT]])
+; CHECK-NEXT: [[NEEDLE_MASKED:%.*]] = and <vscale x 16 x i1> [[TMP2]], [[NEEDLE_PRED]]
+; CHECK-NEXT: [[PNEEDLE:%.*]] = getelementptr i8, ptr null, i64 [[NEEDLE_IDX]]
+; CHECK-NEXT: [[NEEDLE_LOAD_VEC:%.*]] = call <vscale x 16 x i8> @llvm.masked.load.nxv16i8.p0(ptr align 1 [[PNEEDLE]], <vscale x 16 x i1> [[NEEDLE_MASKED]], <vscale x 16 x i8> zeroinitializer)
+; CHECK-NEXT: [[NEEDLE0:%.*]] = extractelement <vscale x 16 x i8> [[NEEDLE_LOAD_VEC]], i64 0
+; CHECK-NEXT: [[NEEDLE0_SPLATINSERT:%.*]] = insertelement <vscale x 16 x i8> poison, i8 [[NEEDLE0]], i64 0
+; CHECK-NEXT: [[NEEDLE0_SPLAT:%.*]] = shufflevector <vscale x 16 x i8> [[NEEDLE0_SPLATINSERT]], <vscale x 16 x i8> poison, <vscale x 16 x i32> zeroinitializer
+; CHECK-NEXT: [[NEEDLE_SPLAT:%.*]] = select <vscale x 16 x i1> [[NEEDLE_MASKED]], <vscale x 16 x i8> [[NEEDLE_LOAD_VEC]], <vscale x 16 x i8> [[NEEDLE0_SPLAT]]
+; CHECK-NEXT: [[NEEDLE_VEC:%.*]] = call <16 x i8> @llvm.vector.extract.v16i8.nxv16i8(<vscale x 16 x i8> [[NEEDLE_SPLAT]], i64 0)
+; CHECK-NEXT: [[MATCH_SEGMENT:%.*]] = call <vscale x 16 x i1> @llvm.experimental.vector.match.nxv16i8.v16i8(<vscale x 16 x i8> [[SEARCH_LOAD_VEC]], <16 x i8> [[NEEDLE_VEC]], <vscale x 16 x i1> [[SEARCH_MASKED]])
+; CHECK-NEXT: [[MATCH_ACCUMULATOR]] = or <vscale x 16 x i1> [[PMATCH]], [[MATCH_SEGMENT]]
+; CHECK-NEXT: [[NEEDLE_IDX_NEXT]] = add i64 [[NEEDLE_IDX]], 16
+; CHECK-NEXT: [[TMP3:%.*]] = icmp ult i64 [[NEEDLE_IDX_NEXT]], [[NEEDLE_TRIP_COUNT]]
+; CHECK-NEXT: br i1 [[TMP3]], label %[[NEEDLE_CHECK_VEC]], label %[[MATCH_CHECK_VEC:.*]]
+; CHECK: [[MATCH_CHECK_VEC]]:
+; CHECK-NEXT: [[MATCH_PRED:%.*]] = phi <vscale x 16 x i1> [ [[MATCH_ACCUMULATOR]], %[[NEEDLE_CHECK_VEC]] ]
+; CHECK-NEXT: [[TMP4:%.*]] = call i1 @llvm.vector.reduce.or.nxv16i1(<vscale x 16 x i1> [[MATCH_PRED]])
+; CHECK-NEXT: br i1 [[TMP4]], label %[[CALCULATE_MATCH:.*]], label %[[SEARCH_CHECK_VEC]]
+; CHECK: [[CALCULATE_MATCH]]:
+; CHECK-NEXT: [[MATCH_START:%.*]] = phi ptr [ [[PSEARCH]], %[[MATCH_CHECK_VEC]] ]
+; CHECK-NEXT: [[MATCH_VEC:%.*]] = phi <vscale x 16 x i1> [ [[MATCH_PRED]], %[[MATCH_CHECK_VEC]] ]
+; CHECK-NEXT: [[MATCH_IDX:%.*]] = call i64 @llvm.experimental.cttz.elts.i64.nxv16i1(<vscale x 16 x i1> [[MATCH_VEC]], i1 true)
+; CHECK-NEXT: [[MATCH_RES:%.*]] = getelementptr i8, ptr [[MATCH_START]], i64 [[MATCH_IDX]]
+; CHECK-NEXT: br label %[[FOUND:.*]]
+; CHECK: [[SEARCH_CHECK_VEC]]:
+; CHECK-NEXT: [[SEARCH_IDX_NEXT]] = add i64 [[SEARCH_IDX]], 16
+; CHECK-NEXT: [[TMP5:%.*]] = icmp ult i64 [[SEARCH_IDX_NEXT]], [[SEARCH_TRIP_COUNT]]
+; CHECK-NEXT: br i1 [[TMP5]], label %[[FIND_FIRST_VEC_HEADER]], label %[[OUTER_LATCH]]
+; CHECK: [[SCALAR_PREHEADER]]:
+; CHECK-NEXT: br label %[[SEARCH_HEADER:.*]]
+; CHECK: [[SEARCH_HEADER]]:
+; CHECK-NEXT: [[S:%.*]] = phi ptr [ null, %[[SCALAR_PREHEADER]] ], [ [[S_NEXT:%.*]], %[[SEARCH_LATCH:.*]] ]
+; CHECK-NEXT: [[SV:%.*]] = load i8, ptr [[S]], align 1
+; CHECK-NEXT: br label %[[NEEDLE_HEADER:.*]]
+; CHECK: [[NEEDLE_LATCH:.*]]:
+; CHECK-NEXT: [[N_NEXT:%.*]] = getelementptr i8, ptr [[N:%.*]], i64 1
+; CHECK-NEXT: [[N_DONE:%.*]] = icmp eq ptr [[N_NEXT]], [[NEEDLE_END]]
+; CHECK-NEXT: br i1 [[N_DONE]], label %[[SEARCH_LATCH]], label %[[NEEDLE_HEADER]]
+; CHECK: [[NEEDLE_HEADER]]:
+; CHECK-NEXT: [[N]] = phi ptr [ null, %[[SEARCH_HEADER]] ], [ [[N_NEXT]], %[[NEEDLE_LATCH]] ]
+; CHECK-NEXT: [[NV:%.*]] = load i8, ptr [[N]], align 1
+; CHECK-NEXT: [[MATCH:%.*]] = icmp eq i8 [[SV]], [[NV]]
+; CHECK-NEXT: br i1 [[MATCH]], label %[[FOUND]], label %[[NEEDLE_LATCH]]
+; CHECK: [[SEARCH_LATCH]]:
+; CHECK-NEXT: [[S_NEXT]] = getelementptr i8, ptr [[S]], i64 1
+; CHECK-NEXT: [[S_DONE:%.*]] = icmp eq ptr [[S_NEXT]], [[SEARCH_END]]
+; CHECK-NEXT: br i1 [[S_DONE]], label %[[OUTER_LATCH]], label %[[SEARCH_HEADER]]
+; CHECK: [[FOUND]]:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %outer.header
+
+outer.latch:
+ br label %outer.header
+
+outer.header:
+ br label %search.header
+
+search.header:
+ %s = phi ptr [ null, %outer.header ], [ %s.next, %search.latch ]
+ %sv = load i8, ptr %s, align 1
+ br label %needle.header
+
+needle.latch:
+ %n.next = getelementptr i8, ptr %n, i64 1
+ %n.done = icmp eq ptr %n.next, %needle_end
+ br i1 %n.done, label %search.latch, label %needle.header
+
+needle.header:
+ %n = phi ptr [ null, %search.header ], [ %n.next, %needle.latch ]
+ %nv = load i8, ptr %n, align 1
+ %match = icmp eq i8 %sv, %nv
+ br i1 %match, label %found, label %needle.latch
+
+search.latch:
+ %s.next = getelementptr i8, ptr %s, i64 1
+ %s.done = icmp eq ptr %s.next, %search_end
+ br i1 %s.done, label %outer.latch, label %search.header
+
+found:
+ ret void
+}
+
+; The same nesting, but a match branches back into the enclosing loop. Here the
+; match-index block does belong to the enclosing loop.
+
+define void @find_first_of_i8_nested_exit_inside(ptr %search_end, ptr %needle_end, ptr %sink) #0 {
+; CHECK-LABEL: define void @find_first_of_i8_nested_exit_inside(
+; CHECK-SAME: ptr [[SEARCH_END:%.*]], ptr [[NEEDLE_END:%.*]], ptr [[SINK:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT: [[ENTRY:.*:]]
+; CHECK-NEXT: br label %[[OUTER_HEADER:.*]]
+; CHECK: [[OUTER_LATCH_LOOPEXIT:.*]]:
+; CHECK-NEXT: br label %[[OUTER_LATCH:.*]]
+; CHECK: [[OUTER_LATCH]]:
+; CHECK-NEXT: br label %[[OUTER_HEADER]]
+; CHECK: [[OUTER_HEADER]]:
+; CHECK-NEXT: br label %[[MEM_CHECK:.*]]
+; CHECK: [[MEM_CHECK]]:
+; CHECK-NEXT: [[SEARCH_END_INT:%.*]] = ptrtoint ptr [[SEARCH_END]] to i64
+; CHECK-NEXT: [[TMP0:%.*]] = ptrtoaddr ptr [[SEARCH_END]] to i64
+; CHECK-NEXT: [[SEARCH_TRIP_COUNT:%.*]] = sub i64 [[TMP0]], 0
+; CHECK-NEXT: [[NEEDLE_END_INT:%.*]] = ptrtoint ptr [[NEEDLE_END]] to i64
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[NEEDLE_END]] to i64
+; CHECK-NEXT: [[NEEDLE_TRIP_COUNT:%.*]] = sub i64 [[TMP1]], 0
+; CHECK-NEXT: [[TMP2:%.*]] = call <vscale x 16 x i1> @llvm.get.active.lane.mask.nxv16i1.i64(i64 0, i64 16)
+; CHECK-NEXT: [[SEARCH_END_PAGE:%.*]] = lshr i64 [[SEARCH_END_INT]], 12
+; CHECK-NEXT: [[NEEDLE_END_PAGE:%.*]] = lshr i64 [[NEEDLE_END_INT]], 12
+; CHECK-NEXT: [[SEARCH_PAGE_CMP:%.*]] = icmp ne i64 0, [[SEARCH_END_PAGE]]
+; CHECK-NEXT: [[NEEDLE_PAGE_CMP:%.*]] = icmp ne i64 0, [[NEEDLE_END_PAGE]]
+; CHECK-NEXT: [[COMBINED_PAGE_CMP:%.*]] = or i1 [[SEARCH_PAGE_CMP]], [[NEEDLE_PAGE_CMP]]
+; CHECK-NEXT: br i1 [[COMBINED_PAGE_CMP]], label %[[SCALAR_PREHEADER:.*]], label %[[FIND_FIRST_VEC_HEADER:.*]], !prof [[PROF0]]
+; CHECK: [[FIND_FIRST_VEC_HEADER]]:
+; CHECK-NEXT: [[SEARCH_IDX:%.*]] = phi i64 [ 0, %[[MEM_CHECK]] ], [ [[SEARCH_IDX_NEXT:%.*]], %[[SEARCH_CHECK_VEC:.*]] ]
+; CHECK-NEXT: [[SEARCH_PRED:%.*]] = call <vscale x 16 x i1> @llvm.get.active.lane.mask.nxv16i1.i64(i64 [[SEARCH_IDX]], i64 [[SEARCH_TRIP_COUNT]])
+; CHECK-NEXT: [[SEARCH_MASKED:%.*]] = and <vscale x 16 x i1> [[TMP2]], [[SEARCH_PRED]]
+; CHECK-NEXT: [[PSEARCH:%.*]] = getelementptr i8, ptr null, i64 [[SEARCH_IDX]]
+; CHECK-NEXT: [[SEARCH_LOAD_VEC:%.*]] = call <vscale x 16 x i8> @llvm.masked.load.nxv16i8.p0(ptr align 1 [[PSEARCH]], <vscale x 16 x i1> [[SEARCH_MASKED]], <vscale x 16 x i8> zeroinitializer)
+; CHECK-NEXT: br label %[[NEEDLE_CHECK_VEC:.*]]
+; CHECK: [[NEEDLE_CHECK_VEC]]:
+; CHECK-NEXT: [[NEEDLE_IDX:%.*]] = phi i64 [ 0, %[[FIND_FIRST_VEC_HEADER]] ], [ [[NEEDLE_IDX_NEXT:%.*]], %[[NEEDLE_CHECK_VEC]] ]
+; CHECK-NEXT: [[PMATCH:%.*]] = phi <vscale x 16 x i1> [ zeroinitializer, %[[FIND_FIRST_VEC_HEADER]] ], [ [[MATCH_ACCUMULATOR:%.*]], %[[NEEDLE_CHECK_VEC]] ]
+; CHECK-NEXT: [[NEEDLE_PRED:%.*]] = call <vscale x 16 x i1> @llvm.get.active.lane.mask.nxv16i1.i64(i64 [[NEEDLE_IDX]], i64 [[NEEDLE_TRIP_COUNT]])
+; CHECK-NEXT: [[NEEDLE_MASKED:%.*]] = and <vscale x 16 x i1> [[TMP2]], [[NEEDLE_PRED]]
+; CHECK-NEXT: [[PNEEDLE:%.*]] = getelementptr i8, ptr null, i64 [[NEEDLE_IDX]]
+; CHECK-NEXT: [[NEEDLE_LOAD_VEC:%.*]] = call <vscale x 16 x i8> @llvm.masked.load.nxv16i8.p0(ptr align 1 [[PNEEDLE]], <vscale x 16 x i1> [[NEEDLE_MASKED]], <vscale x 16 x i8> zeroinitializer)
+; CHECK-NEXT: [[NEEDLE0:%.*]] = extractelement <vscale x 16 x i8> [[NEEDLE_LOAD_VEC]], i64 0
+; CHECK-NEXT: [[NEEDLE0_SPLATINSERT:%.*]] = insertelement <vscale x 16 x i8> poison, i8 [[NEEDLE0]], i64 0
+; CHECK-NEXT: [[NEEDLE0_SPLAT:%.*]] = shufflevector <vscale x 16 x i8> [[NEEDLE0_SPLATINSERT]], <vscale x 16 x i8> poison, <vscale x 16 x i32> zeroinitializer
+; CHECK-NEXT: [[NEEDLE_SPLAT:%.*]] = select <vscale x 16 x i1> [[NEEDLE_MASKED]], <vscale x 16 x i8> [[NEEDLE_LOAD_VEC]], <vscale x 16 x i8> [[NEEDLE0_SPLAT]]
+; CHECK-NEXT: [[NEEDLE_VEC:%.*]] = call <16 x i8> @llvm.vector.extract.v16i8.nxv16i8(<vscale x 16 x i8> [[NEEDLE_SPLAT]], i64 0)
+; CHECK-NEXT: [[MATCH_SEGMENT:%.*]] = call <vscale x 16 x i1> @llvm.experimental.vector.match.nxv16i8.v16i8(<vscale x 16 x i8> [[SEARCH_LOAD_VEC]], <16 x i8> [[NEEDLE_VEC]], <vscale x 16 x i1> [[SEARCH_MASKED]])
+; CHECK-NEXT: [[MATCH_ACCUMULATOR]] = or <vscale x 16 x i1> [[PMATCH]], [[MATCH_SEGMENT]]
+; CHECK-NEXT: [[NEEDLE_IDX_NEXT]] = add i64 [[NEEDLE_IDX]], 16
+; CHECK-NEXT: [[TMP3:%.*]] = icmp ult i64 [[NEEDLE_IDX_NEXT]], [[NEEDLE_TRIP_COUNT]]
+; CHECK-NEXT: br i1 [[TMP3]], label %[[NEEDLE_CHECK_VEC]], label %[[MATCH_CHECK_VEC:.*]]
+; CHECK: [[MATCH_CHECK_VEC]]:
+; CHECK-NEXT: [[MATCH_PRED:%.*]] = phi <vscale x 16 x i1> [ [[MATCH_ACCUMULATOR]], %[[NEEDLE_CHECK_VEC]] ]
+; CHECK-NEXT: [[TMP4:%.*]] = call i1 @llvm.vector.reduce.or.nxv16i1(<vscale x 16 x i1> [[MATCH_PRED]])
+; CHECK-NEXT: br i1 [[TMP4]], label %[[CALCULATE_MATCH:.*]], label %[[SEARCH_CHECK_VEC]]
+; CHECK: [[CALCULATE_MATCH]]:
+; CHECK-NEXT: [[MATCH_START:%.*]] = phi ptr [ [[PSEARCH]], %[[MATCH_CHECK_VEC]] ]
+; CHECK-NEXT: [[MATCH_VEC:%.*]] = phi <vscale x 16 x i1> [ [[MATCH_PRED]], %[[MATCH_CHECK_VEC]] ]
+; CHECK-NEXT: [[MATCH_IDX:%.*]] = call i64 @llvm.experimental.cttz.elts.i64.nxv16i1(<vscale x 16 x i1> [[MATCH_VEC]], i1 true)
+; CHECK-NEXT: [[MATCH_RES:%.*]] = getelementptr i8, ptr [[MATCH_START]], i64 [[MATCH_IDX]]
+; CHECK-NEXT: br label %[[FOUND:.*]]
+; CHECK: [[SEARCH_CHECK_VEC]]:
+; CHECK-NEXT: [[SEARCH_IDX_NEXT]] = add i64 [[SEARCH_IDX]], 16
+; CHECK-NEXT: [[TMP5:%.*]] = icmp ult i64 [[SEARCH_IDX_NEXT]], [[SEARCH_TRIP_COUNT]]
+; CHECK-NEXT: br i1 [[TMP5]], label %[[FIND_FIRST_VEC_HEADER]], label %[[OUTER_LATCH_LOOPEXIT]]
+; CHECK: [[SCALAR_PREHEADER]]:
+; CHECK-NEXT: br label %[[SEARCH_HEADER:.*]]
+; CHECK: [[SEARCH_HEADER]]:
+; CHECK-NEXT: [[S:%.*]] = phi ptr [ null, %[[SCALAR_PREHEADER]] ], [ [[S_NEXT:%.*]], %[[SEARCH_LATCH:.*]] ]
+; CHECK-NEXT: [[SV:%.*]] = load i8, ptr [[S]], align 1
+; CHECK-NEXT: br label %[[NEEDLE_HEADER:.*]]
+; CHECK: [[NEEDLE_LATCH:.*]]:
+; CHECK-NEXT: [[N_NEXT:%.*]] = getelementptr i8, ptr [[N:%.*]], i64 1
+; CHECK-NEXT: [[N_DONE:%.*]] = icmp eq ptr [[N_NEXT]], [[NEEDLE_END]]
+; CHECK-NEXT: br i1 [[N_DONE]], label %[[SEARCH_LATCH]], label %[[NEEDLE_HEADER]]
+; CHECK: [[NEEDLE_HEADER]]:
+; CHECK-NEXT: [[N]] = phi ptr [ null, %[[SEARCH_HEADER]] ], [ [[N_NEXT]], %[[NEEDLE_LATCH]] ]
+; CHECK-NEXT: [[NV:%.*]] = load i8, ptr [[N]], align 1
+; CHECK-NEXT: [[MATCH:%.*]] = icmp eq i8 [[SV]], [[NV]]
+; CHECK-NEXT: br i1 [[MATCH]], label %[[FOUND]], label %[[NEEDLE_LATCH]]
+; CHECK: [[SEARCH_LATCH]]:
+; CHECK-NEXT: [[S_NEXT]] = getelementptr i8, ptr [[S]], i64 1
+; CHECK-NEXT: [[S_DONE:%.*]] = icmp eq ptr [[S_NEXT]], [[SEARCH_END]]
+; CHECK-NEXT: br i1 [[S_DONE]], label %[[OUTER_LATCH_LOOPEXIT]], label %[[SEARCH_HEADER]]
+; CHECK: [[FOUND]]:
+; CHECK-NEXT: store i8 1, ptr [[SINK]], align 1
+; CHECK-NEXT: br label %[[OUTER_LATCH]]
+;
+entry:
+ br label %outer.header
+
+outer.latch:
+ br label %outer.header
+
+outer.header:
+ br label %search.header
+
+search.header:
+ %s = phi ptr [ null, %outer.header ], [ %s.next, %search.latch ]
+ %sv = load i8, ptr %s, align 1
+ br label %needle.header
+
+needle.latch:
+ %n.next = getelementptr i8, ptr %n, i64 1
+ %n.done = icmp eq ptr %n.next, %needle_end
+ br i1 %n.done, label %search.latch, label %needle.header
+
+needle.header:
+ %n = phi ptr [ null, %search.header ], [ %n.next, %needle.latch ]
+ %nv = load i8, ptr %n, align 1
+ %match = icmp eq i8 %sv, %nv
+ br i1 %match, label %found, label %needle.latch
+
+search.latch:
+ %s.next = getelementptr i8, ptr %s, i64 1
+ %s.done = icmp eq ptr %s.next, %search_end
+ br i1 %s.done, label %outer.latch, label %search.header
+
+found:
+ store i8 1, ptr %sink, align 1
+ br label %outer.latch
+}
+
+attributes #0 = { "target-features"="+sve2" }
+;.
+; CHECK: [[PROF0]] = !{!"branch_weights", i32 10, i32 90}
+;.
>From 07d86b20d484e8c9015e6e535d52c7a8436b8d6f Mon Sep 17 00:00:00 2001
From: Kunal Pathak <kupathak at meta.com>
Date: Wed, 23 Sep 2026 10:35:22 -0700
Subject: [PATCH 2/3] address review comments
---
.../Vectorize/LoopIdiomVectorize.cpp | 14 +-
.../AArch64/find-first-byte-nested.ll | 139 ++++++++++++++++++
2 files changed, 148 insertions(+), 5 deletions(-)
diff --git a/llvm/lib/Transforms/Vectorize/LoopIdiomVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopIdiomVectorize.cpp
index 2854e50a2613d8..8316341d830ad3 100644
--- a/llvm/lib/Transforms/Vectorize/LoopIdiomVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopIdiomVectorize.cpp
@@ -1253,15 +1253,19 @@ Value *LoopIdiomVectorize::expandFindFirstByte(
if (auto ParentLoop = CurLoop->getParentLoop()) {
ParentLoop->addBasicBlockToLoop(BB0, *LI);
ParentLoop->addChildLoop(OuterLoop);
- // BB4 branches only to ExitSucc, so it belongs to the parent loop only when
- // that exit is itself inside the parent loop. Otherwise BB4 always leaves,
- // which would leave a parent block with no in-loop successor.
- if (ParentLoop->contains(ExitSucc))
- ParentLoop->addBasicBlockToLoop(BB4, *LI);
} else {
LI->addTopLevelLoop(OuterLoop);
}
+ // BB4 branches only to ExitSucc, so it belongs to the innermost enclosing
+ // loop that contains ExitSucc. That is not necessarily CurLoop's parent: a
+ // match can exit several levels out, or out of every loop.
+ Loop *ExitLoop = CurLoop->getParentLoop();
+ while (ExitLoop && !ExitLoop->contains(ExitSucc))
+ ExitLoop = ExitLoop->getParentLoop();
+ if (ExitLoop)
+ ExitLoop->addBasicBlockToLoop(BB4, *LI);
+
// Add the inner loop to the outer.
OuterLoop->addChildLoop(InnerLoop);
diff --git a/llvm/test/Transforms/LoopIdiom/AArch64/find-first-byte-nested.ll b/llvm/test/Transforms/LoopIdiom/AArch64/find-first-byte-nested.ll
index ff4febad22cef3..bb927ff3922d3e 100644
--- a/llvm/test/Transforms/LoopIdiom/AArch64/find-first-byte-nested.ll
+++ b/llvm/test/Transforms/LoopIdiom/AArch64/find-first-byte-nested.ll
@@ -247,6 +247,145 @@ found:
br label %outer.latch
}
+; Three levels of nesting: a match exits the enclosing loop but lands inside a
+; further enclosing loop. The match-index block belongs to that grandparent
+; loop, not to the immediate parent and not to no loop at all.
+
+define void @find_first_of_i8_nested_exit_in_grandparent(ptr %search_end, ptr %needle_end, ptr %sink) #0 {
+; CHECK-LABEL: define void @find_first_of_i8_nested_exit_in_grandparent(
+; CHECK-SAME: ptr [[SEARCH_END:%.*]], ptr [[NEEDLE_END:%.*]], ptr [[SINK:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT: [[ENTRY:.*:]]
+; CHECK-NEXT: br label %[[GP_HEADER:.*]]
+; CHECK: [[GP_HEADER]]:
+; CHECK-NEXT: br label %[[PARENT_HEADER:.*]]
+; CHECK: [[PARENT_HEADER]]:
+; CHECK-NEXT: br label %[[MEM_CHECK:.*]]
+; CHECK: [[MEM_CHECK]]:
+; CHECK-NEXT: [[SEARCH_END_INT:%.*]] = ptrtoint ptr [[SEARCH_END]] to i64
+; CHECK-NEXT: [[TMP0:%.*]] = ptrtoaddr ptr [[SEARCH_END]] to i64
+; CHECK-NEXT: [[SEARCH_TRIP_COUNT:%.*]] = sub i64 [[TMP0]], 0
+; CHECK-NEXT: [[NEEDLE_END_INT:%.*]] = ptrtoint ptr [[NEEDLE_END]] to i64
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[NEEDLE_END]] to i64
+; CHECK-NEXT: [[NEEDLE_TRIP_COUNT:%.*]] = sub i64 [[TMP1]], 0
+; CHECK-NEXT: [[TMP2:%.*]] = call <vscale x 16 x i1> @llvm.get.active.lane.mask.nxv16i1.i64(i64 0, i64 16)
+; CHECK-NEXT: [[SEARCH_END_PAGE:%.*]] = lshr i64 [[SEARCH_END_INT]], 12
+; CHECK-NEXT: [[NEEDLE_END_PAGE:%.*]] = lshr i64 [[NEEDLE_END_INT]], 12
+; CHECK-NEXT: [[SEARCH_PAGE_CMP:%.*]] = icmp ne i64 0, [[SEARCH_END_PAGE]]
+; CHECK-NEXT: [[NEEDLE_PAGE_CMP:%.*]] = icmp ne i64 0, [[NEEDLE_END_PAGE]]
+; CHECK-NEXT: [[COMBINED_PAGE_CMP:%.*]] = or i1 [[SEARCH_PAGE_CMP]], [[NEEDLE_PAGE_CMP]]
+; CHECK-NEXT: br i1 [[COMBINED_PAGE_CMP]], label %[[SCALAR_PREHEADER:.*]], label %[[FIND_FIRST_VEC_HEADER:.*]], !prof [[PROF0]]
+; CHECK: [[FIND_FIRST_VEC_HEADER]]:
+; CHECK-NEXT: [[SEARCH_IDX:%.*]] = phi i64 [ 0, %[[MEM_CHECK]] ], [ [[SEARCH_IDX_NEXT:%.*]], %[[SEARCH_CHECK_VEC:.*]] ]
+; CHECK-NEXT: [[SEARCH_PRED:%.*]] = call <vscale x 16 x i1> @llvm.get.active.lane.mask.nxv16i1.i64(i64 [[SEARCH_IDX]], i64 [[SEARCH_TRIP_COUNT]])
+; CHECK-NEXT: [[SEARCH_MASKED:%.*]] = and <vscale x 16 x i1> [[TMP2]], [[SEARCH_PRED]]
+; CHECK-NEXT: [[PSEARCH:%.*]] = getelementptr i8, ptr null, i64 [[SEARCH_IDX]]
+; CHECK-NEXT: [[SEARCH_LOAD_VEC:%.*]] = call <vscale x 16 x i8> @llvm.masked.load.nxv16i8.p0(ptr align 1 [[PSEARCH]], <vscale x 16 x i1> [[SEARCH_MASKED]], <vscale x 16 x i8> zeroinitializer)
+; CHECK-NEXT: br label %[[NEEDLE_CHECK_VEC:.*]]
+; CHECK: [[NEEDLE_CHECK_VEC]]:
+; CHECK-NEXT: [[NEEDLE_IDX:%.*]] = phi i64 [ 0, %[[FIND_FIRST_VEC_HEADER]] ], [ [[NEEDLE_IDX_NEXT:%.*]], %[[NEEDLE_CHECK_VEC]] ]
+; CHECK-NEXT: [[PMATCH:%.*]] = phi <vscale x 16 x i1> [ zeroinitializer, %[[FIND_FIRST_VEC_HEADER]] ], [ [[MATCH_ACCUMULATOR:%.*]], %[[NEEDLE_CHECK_VEC]] ]
+; CHECK-NEXT: [[NEEDLE_PRED:%.*]] = call <vscale x 16 x i1> @llvm.get.active.lane.mask.nxv16i1.i64(i64 [[NEEDLE_IDX]], i64 [[NEEDLE_TRIP_COUNT]])
+; CHECK-NEXT: [[NEEDLE_MASKED:%.*]] = and <vscale x 16 x i1> [[TMP2]], [[NEEDLE_PRED]]
+; CHECK-NEXT: [[PNEEDLE:%.*]] = getelementptr i8, ptr null, i64 [[NEEDLE_IDX]]
+; CHECK-NEXT: [[NEEDLE_LOAD_VEC:%.*]] = call <vscale x 16 x i8> @llvm.masked.load.nxv16i8.p0(ptr align 1 [[PNEEDLE]], <vscale x 16 x i1> [[NEEDLE_MASKED]], <vscale x 16 x i8> zeroinitializer)
+; CHECK-NEXT: [[NEEDLE0:%.*]] = extractelement <vscale x 16 x i8> [[NEEDLE_LOAD_VEC]], i64 0
+; CHECK-NEXT: [[NEEDLE0_SPLATINSERT:%.*]] = insertelement <vscale x 16 x i8> poison, i8 [[NEEDLE0]], i64 0
+; CHECK-NEXT: [[NEEDLE0_SPLAT:%.*]] = shufflevector <vscale x 16 x i8> [[NEEDLE0_SPLATINSERT]], <vscale x 16 x i8> poison, <vscale x 16 x i32> zeroinitializer
+; CHECK-NEXT: [[NEEDLE_SPLAT:%.*]] = select <vscale x 16 x i1> [[NEEDLE_MASKED]], <vscale x 16 x i8> [[NEEDLE_LOAD_VEC]], <vscale x 16 x i8> [[NEEDLE0_SPLAT]]
+; CHECK-NEXT: [[NEEDLE_VEC:%.*]] = call <16 x i8> @llvm.vector.extract.v16i8.nxv16i8(<vscale x 16 x i8> [[NEEDLE_SPLAT]], i64 0)
+; CHECK-NEXT: [[MATCH_SEGMENT:%.*]] = call <vscale x 16 x i1> @llvm.experimental.vector.match.nxv16i8.v16i8(<vscale x 16 x i8> [[SEARCH_LOAD_VEC]], <16 x i8> [[NEEDLE_VEC]], <vscale x 16 x i1> [[SEARCH_MASKED]])
+; CHECK-NEXT: [[MATCH_ACCUMULATOR]] = or <vscale x 16 x i1> [[PMATCH]], [[MATCH_SEGMENT]]
+; CHECK-NEXT: [[NEEDLE_IDX_NEXT]] = add i64 [[NEEDLE_IDX]], 16
+; CHECK-NEXT: [[TMP3:%.*]] = icmp ult i64 [[NEEDLE_IDX_NEXT]], [[NEEDLE_TRIP_COUNT]]
+; CHECK-NEXT: br i1 [[TMP3]], label %[[NEEDLE_CHECK_VEC]], label %[[MATCH_CHECK_VEC:.*]]
+; CHECK: [[MATCH_CHECK_VEC]]:
+; CHECK-NEXT: [[MATCH_PRED:%.*]] = phi <vscale x 16 x i1> [ [[MATCH_ACCUMULATOR]], %[[NEEDLE_CHECK_VEC]] ]
+; CHECK-NEXT: [[TMP4:%.*]] = call i1 @llvm.vector.reduce.or.nxv16i1(<vscale x 16 x i1> [[MATCH_PRED]])
+; CHECK-NEXT: br i1 [[TMP4]], label %[[CALCULATE_MATCH:.*]], label %[[SEARCH_CHECK_VEC]]
+; CHECK: [[CALCULATE_MATCH]]:
+; CHECK-NEXT: [[MATCH_START:%.*]] = phi ptr [ [[PSEARCH]], %[[MATCH_CHECK_VEC]] ]
+; CHECK-NEXT: [[MATCH_VEC:%.*]] = phi <vscale x 16 x i1> [ [[MATCH_PRED]], %[[MATCH_CHECK_VEC]] ]
+; CHECK-NEXT: [[MATCH_IDX:%.*]] = call i64 @llvm.experimental.cttz.elts.i64.nxv16i1(<vscale x 16 x i1> [[MATCH_VEC]], i1 true)
+; CHECK-NEXT: [[MATCH_RES:%.*]] = getelementptr i8, ptr [[MATCH_START]], i64 [[MATCH_IDX]]
+; CHECK-NEXT: br label %[[FOUND:.*]]
+; CHECK: [[SEARCH_CHECK_VEC]]:
+; CHECK-NEXT: [[SEARCH_IDX_NEXT]] = add i64 [[SEARCH_IDX]], 16
+; CHECK-NEXT: [[TMP5:%.*]] = icmp ult i64 [[SEARCH_IDX_NEXT]], [[SEARCH_TRIP_COUNT]]
+; CHECK-NEXT: br i1 [[TMP5]], label %[[FIND_FIRST_VEC_HEADER]], label %[[PARENT_LATCH:.*]]
+; CHECK: [[SCALAR_PREHEADER]]:
+; CHECK-NEXT: br label %[[SEARCH_HEADER:.*]]
+; CHECK: [[SEARCH_HEADER]]:
+; CHECK-NEXT: [[S:%.*]] = phi ptr [ null, %[[SCALAR_PREHEADER]] ], [ [[S_NEXT:%.*]], %[[SEARCH_LATCH:.*]] ]
+; CHECK-NEXT: [[SV:%.*]] = load i8, ptr [[S]], align 1
+; CHECK-NEXT: br label %[[NEEDLE_HEADER:.*]]
+; CHECK: [[NEEDLE_HEADER]]:
+; CHECK-NEXT: [[N:%.*]] = phi ptr [ null, %[[SEARCH_HEADER]] ], [ [[N_NEXT:%.*]], %[[NEEDLE_LATCH:.*]] ]
+; CHECK-NEXT: [[NV:%.*]] = load i8, ptr [[N]], align 1
+; CHECK-NEXT: [[MATCH:%.*]] = icmp eq i8 [[SV]], [[NV]]
+; CHECK-NEXT: br i1 [[MATCH]], label %[[FOUND]], label %[[NEEDLE_LATCH]]
+; CHECK: [[NEEDLE_LATCH]]:
+; CHECK-NEXT: [[N_NEXT]] = getelementptr i8, ptr [[N]], i64 1
+; CHECK-NEXT: [[N_DONE:%.*]] = icmp eq ptr [[N_NEXT]], [[NEEDLE_END]]
+; CHECK-NEXT: br i1 [[N_DONE]], label %[[SEARCH_LATCH]], label %[[NEEDLE_HEADER]]
+; CHECK: [[SEARCH_LATCH]]:
+; CHECK-NEXT: [[S_NEXT]] = getelementptr i8, ptr [[S]], i64 1
+; CHECK-NEXT: [[S_DONE:%.*]] = icmp eq ptr [[S_NEXT]], [[SEARCH_END]]
+; CHECK-NEXT: br i1 [[S_DONE]], label %[[PARENT_LATCH]], label %[[SEARCH_HEADER]]
+; CHECK: [[PARENT_LATCH]]:
+; CHECK-NEXT: [[PC:%.*]] = load i8, ptr [[SINK]], align 1
+; CHECK-NEXT: [[PD:%.*]] = icmp eq i8 [[PC]], 0
+; CHECK-NEXT: br i1 [[PD]], label %[[PARENT_HEADER]], label %[[GP_LATCH_LOOPEXIT:.*]]
+; CHECK: [[FOUND]]:
+; CHECK-NEXT: store i8 1, ptr [[SINK]], align 1
+; CHECK-NEXT: br label %[[GP_LATCH:.*]]
+; CHECK: [[GP_LATCH_LOOPEXIT]]:
+; CHECK-NEXT: br label %[[GP_LATCH]]
+; CHECK: [[GP_LATCH]]:
+; CHECK-NEXT: br label %[[GP_HEADER]]
+;
+entry:
+ br label %gp.header
+
+gp.header:
+ br label %parent.header
+
+parent.header:
+ br label %search.header
+
+search.header:
+ %s = phi ptr [ null, %parent.header ], [ %s.next, %search.latch ]
+ %sv = load i8, ptr %s, align 1
+ br label %needle.header
+
+needle.header:
+ %n = phi ptr [ null, %search.header ], [ %n.next, %needle.latch ]
+ %nv = load i8, ptr %n, align 1
+ %match = icmp eq i8 %sv, %nv
+ br i1 %match, label %found, label %needle.latch
+
+needle.latch:
+ %n.next = getelementptr i8, ptr %n, i64 1
+ %n.done = icmp eq ptr %n.next, %needle_end
+ br i1 %n.done, label %search.latch, label %needle.header
+
+search.latch:
+ %s.next = getelementptr i8, ptr %s, i64 1
+ %s.done = icmp eq ptr %s.next, %search_end
+ br i1 %s.done, label %parent.latch, label %search.header
+
+parent.latch:
+ %pc = load i8, ptr %sink, align 1
+ %pd = icmp eq i8 %pc, 0
+ br i1 %pd, label %parent.header, label %gp.latch
+
+found:
+ store i8 1, ptr %sink, align 1
+ br label %gp.latch
+
+gp.latch:
+ br label %gp.header
+}
+
attributes #0 = { "target-features"="+sve2" }
;.
; CHECK: [[PROF0]] = !{!"branch_weights", i32 10, i32 90}
>From dce7a8d16922a666d771035f500d3331bdc1c30b Mon Sep 17 00:00:00 2001
From: Kunal Pathak <kupathak at meta.com>
Date: Thu, 24 Sep 2026 12:06:24 -0700
Subject: [PATCH 3/3] Add loop data in test
---
.../LoopIdiom/AArch64/find-first-byte-nested.ll | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/llvm/test/Transforms/LoopIdiom/AArch64/find-first-byte-nested.ll b/llvm/test/Transforms/LoopIdiom/AArch64/find-first-byte-nested.ll
index bb927ff3922d3e..db6706756daca3 100644
--- a/llvm/test/Transforms/LoopIdiom/AArch64/find-first-byte-nested.ll
+++ b/llvm/test/Transforms/LoopIdiom/AArch64/find-first-byte-nested.ll
@@ -1,5 +1,19 @@
; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
; RUN: opt -mtriple=aarch64 -mattr=+sve -passes='loop(loop-idiom-vectorize)' -verify-loop-info -verify-dom-info -S < %s | FileCheck %s
+; RUN: opt -mtriple=aarch64 -mattr=+sve -passes='loop(loop-idiom-vectorize),verify<loops>,print<loops>' -disable-output < %s 2>&1 | FileCheck %s --check-prefix=LOOPS
+
+; Where the expansion's new blocks land in the loop nest. %calculate_match
+; computes the match index and branches only to the successful exit, so it
+; belongs to whichever loop holds that exit: no loop in the first function,
+; the parent in the second, the grandparent but NOT the parent in the third.
+; LOOPS-LABEL: Loop info for function 'find_first_of_i8_nested_exit_outside':
+; LOOPS-NEXT: Loop at depth 1 containing: %outer.header<header>,%search.header,%needle.header<exiting>,%needle.latch,%search.latch,%outer.latch<latch>,%scalar_preheader,%mem_check,%find_first_vec_header,%match_check_vec<exiting>,%search_check_vec,%needle_check_vec
+; LOOPS-LABEL: Loop info for function 'find_first_of_i8_nested_exit_inside':
+; LOOPS-NEXT: Loop at depth 1 containing: %outer.header<header>,%search.header,%needle.header,%needle.latch,%search.latch,%found,%outer.latch<latch>,%outer.latch.loopexit,%scalar_preheader,%mem_check,%calculate_match,%find_first_vec_header,%match_check_vec,%search_check_vec,%needle_check_vec
+; LOOPS-LABEL: Loop info for function 'find_first_of_i8_nested_exit_in_grandparent':
+; LOOPS-NEXT: Loop at depth 1 containing: %gp.header<header>,%parent.header,%search.header,%needle.header,%needle.latch,%search.latch,%parent.latch,%found,%gp.latch<latch>,%gp.latch.loopexit,%scalar_preheader,%mem_check,%calculate_match,%find_first_vec_header,%match_check_vec,%search_check_vec,%needle_check_vec
+; LOOPS-NEXT: Loop at depth 2 containing: %parent.header<header>,%search.header,%needle.header<exiting>,%needle.latch,%search.latch,%parent.latch<latch><exiting>,%scalar_preheader,%mem_check,%find_first_vec_header,%match_check_vec<exiting>,%search_check_vec,%needle_check_vec
+
; The find_first_of idiom nested inside an enclosing loop. The expansion adds
; the block that computes the match index to the enclosing loop, but that block
@@ -390,3 +404,5 @@ attributes #0 = { "target-features"="+sve2" }
;.
; CHECK: [[PROF0]] = !{!"branch_weights", i32 10, i32 90}
;.
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; LOOPS: {{.*}}
More information about the llvm-commits
mailing list