[llvm] b33fbba - Reland [SimplifyCFG] FoldBranchToCommonDest: lift use-restriction on bonus instructions
Roman Lebedev via llvm-commits
llvm-commits at lists.llvm.org
Fri Nov 27 01:48:02 PST 2020
Author: Roman Lebedev
Date: 2020-11-27T12:47:15+03:00
New Revision: b33fbbaa34f0fe9fb16789afc72ae424c1825b69
URL: https://github.com/llvm/llvm-project/commit/b33fbbaa34f0fe9fb16789afc72ae424c1825b69
DIFF: https://github.com/llvm/llvm-project/commit/b33fbbaa34f0fe9fb16789afc72ae424c1825b69.diff
LOG: Reland [SimplifyCFG] FoldBranchToCommonDest: lift use-restriction on bonus instructions
This was orginally committed in 2245fb8aaa1c1f85f53f7b19a1ee3ac69b1a1dfe.
but was immediately reverted in f3abd54958ab90ba7c100d3fa936a3ce0dd2ad04
because of a PHI handling issue.
Original commit message:
1. It doesn't make sense to enforce that the bonus instruction
is only used once in it's basic block. What matters is
whether those user instructions fit within our budget, sure,
but that is another question.
2. It doesn't make sense to enforce that said bonus instructions
are only used within their basic block. Perhaps the branch
condition isn't using the value computed by said bonus instruction,
and said bonus instruction is simply being calculated
to be used in successors?
So iff we can clone bonus instructions, to lift these restrictions,
we just need to carefully update their external uses
to use the new cloned instructions.
Notably, this transform (even without this change) appears to be
poison-unsafe as per alive2, but is otherwise (including the patch) legal.
We don't introduce any new PHI nodes, but only "move" the instructions
around, i'm not really seeing much potential for extra cost modelling
for the transform, especially since now we allow at most one such
bonus instruction by default.
This causes the fold to fire +11.4% more (13216 -> 14725)
as of vanilla llvm test-suite + RawSpeed.
The motivational pattern is IEEE-754-2008 Binary16->Binary32
extension code:
https://github.com/darktable-org/rawspeed/blob/ca57d77fb2ba81f21fc712cfac26e54f46406473/src/librawspeed/common/FloatingPoint.h#L115-L120
^ that should be a switch, but it is not now: https://godbolt.org/z/bvja5v
That being said, even thought this seemed like this would fix it: https://godbolt.org/z/xGq3TM
apparently that fold is happening somewhere else afterall,
so something else also has a similar 'artificial' restriction.
Added:
Modified:
llvm/lib/Transforms/Utils/SimplifyCFG.cpp
llvm/test/CodeGen/Thumb2/mve-float16regloops.ll
llvm/test/CodeGen/Thumb2/mve-float32regloops.ll
llvm/test/CodeGen/Thumb2/mve-postinc-lsr.ll
llvm/test/Transforms/LoopUnroll/peel-loop-inner.ll
llvm/test/Transforms/SimplifyCFG/fold-branch-to-common-dest.ll
Removed:
################################################################################
diff --git a/llvm/lib/Transforms/Utils/SimplifyCFG.cpp b/llvm/lib/Transforms/Utils/SimplifyCFG.cpp
index 0f4c1b4038e5..55ce05987bfd 100644
--- a/llvm/lib/Transforms/Utils/SimplifyCFG.cpp
+++ b/llvm/lib/Transforms/Utils/SimplifyCFG.cpp
@@ -2779,15 +2779,9 @@ bool llvm::FoldBranchToCommonDest(BranchInst *BI, MemorySSAUpdater *MSSAU,
// Ignore dbg intrinsics.
if (isa<DbgInfoIntrinsic>(I))
continue;
- if (!I->hasOneUse() || !isSafeToSpeculativelyExecute(&*I))
+ // I must be safe to execute unconditionally.
+ if (!isSafeToSpeculativelyExecute(&*I))
return Changed;
- // I has only one use and can be executed unconditionally.
- Instruction *User = dyn_cast<Instruction>(I->user_back());
- if (User == nullptr || User->getParent() != BB)
- return Changed;
- // I is used in the same BB. Since BI uses Cond and doesn't have more slots
- // to use any other instruction, User must be an instruction between next(I)
- // and Cond.
// Account for the cost of duplicating this instruction into each
// predecessor.
@@ -2883,6 +2877,13 @@ bool llvm::FoldBranchToCommonDest(BranchInst *BI, MemorySSAUpdater *MSSAU,
PBI->swapSuccessors();
}
+ // Before cloning instructions, notify the successor basic block that it
+ // is about to have a new predecessor. This will update PHI nodes,
+ // which will allow us to update live-out uses of bonus instructions.
+ if (BI->isConditional())
+ AddPredecessorToBlock(PBI->getSuccessor(0) == BB ? TrueDest : FalseDest,
+ PredBlock, BB, MSSAU);
+
// If we have bonus instructions, clone them into the predecessor block.
// Note that there may be multiple predecessor blocks, so we cannot move
// bonus instructions to a predecessor block.
@@ -2914,6 +2915,18 @@ bool llvm::FoldBranchToCommonDest(BranchInst *BI, MemorySSAUpdater *MSSAU,
PredBlock->getInstList().insert(PBI->getIterator(), NewBonusInst);
NewBonusInst->takeName(&*BonusInst);
BonusInst->setName(BonusInst->getName() + ".old");
+ BonusInst->replaceUsesWithIf(NewBonusInst, [BB](Use &U) {
+ auto *User = cast<Instruction>(U.getUser());
+ // Ignore uses in the same block as the bonus instruction itself.
+ if (User->getParent() == BB)
+ return false;
+ // We can safely update external non-PHI uses.
+ if (!isa<PHINode>(User))
+ return true;
+ // For PHI nodes, don't touch incoming values for same block
+ // as the bonus instruction itself.
+ return cast<PHINode>(User)->getIncomingBlock(U) != BB;
+ });
}
// Clone Cond into the predecessor basic block, and or/and the
@@ -2955,7 +2968,6 @@ bool llvm::FoldBranchToCommonDest(BranchInst *BI, MemorySSAUpdater *MSSAU,
(SuccFalseWeight + SuccTrueWeight) +
PredTrueWeight * SuccFalseWeight);
}
- AddPredecessorToBlock(TrueDest, PredBlock, BB, MSSAU);
PBI->setSuccessor(0, TrueDest);
}
if (PBI->getSuccessor(1) == BB) {
@@ -2970,7 +2982,6 @@ bool llvm::FoldBranchToCommonDest(BranchInst *BI, MemorySSAUpdater *MSSAU,
// FalseWeight is FalseWeight for PBI * FalseWeight for BI.
NewWeights.push_back(PredFalseWeight * SuccFalseWeight);
}
- AddPredecessorToBlock(FalseDest, PredBlock, BB, MSSAU);
PBI->setSuccessor(1, FalseDest);
}
if (NewWeights.size() == 2) {
diff --git a/llvm/test/CodeGen/Thumb2/mve-float16regloops.ll b/llvm/test/CodeGen/Thumb2/mve-float16regloops.ll
index 5bf4ebf92f14..fa58c72a0efb 100644
--- a/llvm/test/CodeGen/Thumb2/mve-float16regloops.ll
+++ b/llvm/test/CodeGen/Thumb2/mve-float16regloops.ll
@@ -1077,36 +1077,34 @@ define void @fir(%struct.arm_fir_instance_f32* nocapture readonly %S, half* noca
; CHECK-NEXT: sub sp, #24
; CHECK-NEXT: cmp r3, #8
; CHECK-NEXT: blo.w .LBB16_12
-; CHECK-NEXT: @ %bb.1: @ %if.then
-; CHECK-NEXT: movs r7, #0
-; CHECK-NEXT: cmp.w r7, r3, lsr #2
+; CHECK-NEXT: @ %bb.1: @ %entry
+; CHECK-NEXT: lsrs.w r12, r3, #2
; CHECK-NEXT: beq.w .LBB16_12
; CHECK-NEXT: @ %bb.2: @ %while.body.lr.ph
; CHECK-NEXT: ldrh r4, [r0]
-; CHECK-NEXT: lsr.w r9, r3, #2
-; CHECK-NEXT: ldrd r5, r12, [r0, #4]
-; CHECK-NEXT: movs r3, #1
+; CHECK-NEXT: movs r6, #1
+; CHECK-NEXT: ldrd r5, r3, [r0, #4]
; CHECK-NEXT: sub.w r0, r4, #8
; CHECK-NEXT: and r8, r0, #7
; CHECK-NEXT: add.w r7, r0, r0, lsr #29
-; CHECK-NEXT: asrs r6, r7, #3
-; CHECK-NEXT: cmp r6, #1
+; CHECK-NEXT: asr.w lr, r7, #3
+; CHECK-NEXT: cmp.w lr, #1
; CHECK-NEXT: it gt
-; CHECK-NEXT: asrgt r3, r7, #3
+; CHECK-NEXT: asrgt r6, r7, #3
; CHECK-NEXT: add.w r7, r5, r4, lsl #1
-; CHECK-NEXT: str r3, [sp] @ 4-byte Spill
-; CHECK-NEXT: subs r3, r7, #2
-; CHECK-NEXT: str r3, [sp, #20] @ 4-byte Spill
-; CHECK-NEXT: rsbs r3, r4, #0
-; CHECK-NEXT: str r3, [sp, #8] @ 4-byte Spill
-; CHECK-NEXT: add.w r3, r12, #16
+; CHECK-NEXT: subs r7, #2
+; CHECK-NEXT: str r7, [sp, #20] @ 4-byte Spill
+; CHECK-NEXT: rsbs r7, r4, #0
+; CHECK-NEXT: str r7, [sp, #8] @ 4-byte Spill
+; CHECK-NEXT: add.w r7, r3, #16
+; CHECK-NEXT: str r6, [sp] @ 4-byte Spill
; CHECK-NEXT: str r4, [sp, #12] @ 4-byte Spill
-; CHECK-NEXT: str r3, [sp, #4] @ 4-byte Spill
+; CHECK-NEXT: str r7, [sp, #4] @ 4-byte Spill
; CHECK-NEXT: b .LBB16_4
; CHECK-NEXT: .LBB16_3: @ %while.end
; CHECK-NEXT: @ in Loop: Header=BB16_4 Depth=1
; CHECK-NEXT: ldr r0, [sp, #8] @ 4-byte Reload
-; CHECK-NEXT: subs.w r9, r9, #1
+; CHECK-NEXT: subs.w r12, r12, #1
; CHECK-NEXT: ldr r1, [sp, #16] @ 4-byte Reload
; CHECK-NEXT: vstrb.8 q0, [r2], #8
; CHECK-NEXT: add.w r0, r5, r0, lsl #1
@@ -1117,16 +1115,16 @@ define void @fir(%struct.arm_fir_instance_f32* nocapture readonly %S, half* noca
; CHECK-NEXT: @ Child Loop BB16_6 Depth 2
; CHECK-NEXT: @ Child Loop BB16_10 Depth 2
; CHECK-NEXT: vldrw.u32 q0, [r1], #8
-; CHECK-NEXT: ldrh.w lr, [r12, #14]
-; CHECK-NEXT: ldrh.w r0, [r12, #12]
+; CHECK-NEXT: ldrh.w lr, [r3, #14]
+; CHECK-NEXT: ldrh r0, [r3, #12]
; CHECK-NEXT: str r1, [sp, #16] @ 4-byte Spill
; CHECK-NEXT: ldr r1, [sp, #20] @ 4-byte Reload
-; CHECK-NEXT: ldrh.w r4, [r12, #10]
-; CHECK-NEXT: ldrh.w r7, [r12, #8]
-; CHECK-NEXT: ldrh.w r3, [r12, #6]
-; CHECK-NEXT: ldrh.w r6, [r12, #4]
-; CHECK-NEXT: ldrh.w r11, [r12, #2]
-; CHECK-NEXT: ldrh.w r10, [r12]
+; CHECK-NEXT: ldrh r4, [r3, #10]
+; CHECK-NEXT: ldrh r7, [r3, #8]
+; CHECK-NEXT: ldrh r6, [r3, #6]
+; CHECK-NEXT: ldrh.w r9, [r3, #4]
+; CHECK-NEXT: ldrh.w r11, [r3, #2]
+; CHECK-NEXT: ldrh.w r10, [r3]
; CHECK-NEXT: vstrb.8 q0, [r1], #8
; CHECK-NEXT: vldrw.u32 q0, [r5]
; CHECK-NEXT: str r1, [sp, #20] @ 4-byte Spill
@@ -1136,10 +1134,10 @@ define void @fir(%struct.arm_fir_instance_f32* nocapture readonly %S, half* noca
; CHECK-NEXT: adds r1, r5, #6
; CHECK-NEXT: vfma.f16 q0, q1, r11
; CHECK-NEXT: vldrw.u32 q1, [r5, #4]
-; CHECK-NEXT: vfma.f16 q0, q1, r6
+; CHECK-NEXT: vfma.f16 q0, q1, r9
; CHECK-NEXT: vldrw.u32 q1, [r1]
; CHECK-NEXT: add.w r1, r5, #10
-; CHECK-NEXT: vfma.f16 q0, q1, r3
+; CHECK-NEXT: vfma.f16 q0, q1, r6
; CHECK-NEXT: vldrw.u32 q1, [r5, #8]
; CHECK-NEXT: vfma.f16 q0, q1, r7
; CHECK-NEXT: vldrw.u32 q1, [r1]
diff --git a/llvm/test/CodeGen/Thumb2/mve-float32regloops.ll b/llvm/test/CodeGen/Thumb2/mve-float32regloops.ll
index 694ba14f84f1..b0fa52b2b162 100644
--- a/llvm/test/CodeGen/Thumb2/mve-float32regloops.ll
+++ b/llvm/test/CodeGen/Thumb2/mve-float32regloops.ll
@@ -1049,36 +1049,34 @@ define void @fir(%struct.arm_fir_instance_f32* nocapture readonly %S, float* noc
; CHECK-NEXT: sub sp, #32
; CHECK-NEXT: cmp r3, #8
; CHECK-NEXT: blo.w .LBB16_12
-; CHECK-NEXT: @ %bb.1: @ %if.then
-; CHECK-NEXT: movs r7, #0
-; CHECK-NEXT: cmp.w r7, r3, lsr #2
+; CHECK-NEXT: @ %bb.1: @ %entry
+; CHECK-NEXT: lsrs.w r12, r3, #2
; CHECK-NEXT: beq.w .LBB16_12
; CHECK-NEXT: @ %bb.2: @ %while.body.lr.ph
-; CHECK-NEXT: ldrh r4, [r0]
-; CHECK-NEXT: lsr.w r10, r3, #2
-; CHECK-NEXT: ldrd r5, r12, [r0, #4]
-; CHECK-NEXT: movs r3, #1
-; CHECK-NEXT: sub.w r7, r4, #8
-; CHECK-NEXT: add.w r0, r7, r7, lsr #29
-; CHECK-NEXT: asrs r6, r0, #3
-; CHECK-NEXT: cmp r6, #1
+; CHECK-NEXT: ldrh r6, [r0]
+; CHECK-NEXT: movs r4, #1
+; CHECK-NEXT: ldrd r5, r10, [r0, #4]
+; CHECK-NEXT: sub.w r3, r6, #8
+; CHECK-NEXT: add.w r0, r3, r3, lsr #29
+; CHECK-NEXT: asrs r7, r0, #3
+; CHECK-NEXT: cmp r7, #1
; CHECK-NEXT: it gt
-; CHECK-NEXT: asrgt r3, r0, #3
-; CHECK-NEXT: add.w r0, r5, r4, lsl #2
+; CHECK-NEXT: asrgt r4, r0, #3
+; CHECK-NEXT: add.w r0, r5, r6, lsl #2
; CHECK-NEXT: sub.w r9, r0, #4
-; CHECK-NEXT: rsbs r0, r4, #0
-; CHECK-NEXT: str r3, [sp, #4] @ 4-byte Spill
-; CHECK-NEXT: and r3, r7, #7
+; CHECK-NEXT: rsbs r0, r6, #0
+; CHECK-NEXT: str r4, [sp, #4] @ 4-byte Spill
+; CHECK-NEXT: and r4, r3, #7
; CHECK-NEXT: str r0, [sp, #16] @ 4-byte Spill
-; CHECK-NEXT: add.w r0, r12, #32
-; CHECK-NEXT: str r4, [sp, #20] @ 4-byte Spill
+; CHECK-NEXT: add.w r0, r10, #32
+; CHECK-NEXT: str r6, [sp, #20] @ 4-byte Spill
; CHECK-NEXT: str r0, [sp, #8] @ 4-byte Spill
-; CHECK-NEXT: str r3, [sp, #12] @ 4-byte Spill
+; CHECK-NEXT: str r4, [sp, #12] @ 4-byte Spill
; CHECK-NEXT: b .LBB16_4
; CHECK-NEXT: .LBB16_3: @ %while.end
; CHECK-NEXT: @ in Loop: Header=BB16_4 Depth=1
; CHECK-NEXT: ldr r0, [sp, #16] @ 4-byte Reload
-; CHECK-NEXT: subs.w r10, r10, #1
+; CHECK-NEXT: subs.w r12, r12, #1
; CHECK-NEXT: vstrb.8 q0, [r2], #16
; CHECK-NEXT: add.w r0, r5, r0, lsl #2
; CHECK-NEXT: add.w r5, r0, #16
@@ -1087,25 +1085,25 @@ define void @fir(%struct.arm_fir_instance_f32* nocapture readonly %S, float* noc
; CHECK-NEXT: @ =>This Loop Header: Depth=1
; CHECK-NEXT: @ Child Loop BB16_6 Depth 2
; CHECK-NEXT: @ Child Loop BB16_10 Depth 2
+; CHECK-NEXT: add.w lr, r10, #8
; CHECK-NEXT: vldrw.u32 q0, [r1], #16
-; CHECK-NEXT: ldrd r7, r6, [r12]
-; CHECK-NEXT: ldrd r0, r4, [r12, #8]
-; CHECK-NEXT: ldrd r3, lr, [r12, #16]
-; CHECK-NEXT: ldrd r11, r8, [r12, #24]
+; CHECK-NEXT: ldrd r3, r7, [r10]
+; CHECK-NEXT: ldm.w lr, {r0, r4, r6, lr}
+; CHECK-NEXT: ldrd r11, r8, [r10, #24]
; CHECK-NEXT: vstrb.8 q0, [r9], #16
; CHECK-NEXT: vldrw.u32 q0, [r5], #32
; CHECK-NEXT: strd r9, r1, [sp, #24] @ 8-byte Folded Spill
; CHECK-NEXT: vldrw.u32 q1, [r5, #-28]
-; CHECK-NEXT: vmul.f32 q0, q0, r7
+; CHECK-NEXT: vmul.f32 q0, q0, r3
; CHECK-NEXT: vldrw.u32 q6, [r5, #-24]
; CHECK-NEXT: vldrw.u32 q4, [r5, #-20]
-; CHECK-NEXT: vfma.f32 q0, q1, r6
+; CHECK-NEXT: vfma.f32 q0, q1, r7
; CHECK-NEXT: vldrw.u32 q5, [r5, #-16]
; CHECK-NEXT: vfma.f32 q0, q6, r0
; CHECK-NEXT: vldrw.u32 q2, [r5, #-12]
; CHECK-NEXT: vfma.f32 q0, q4, r4
; CHECK-NEXT: vldrw.u32 q3, [r5, #-8]
-; CHECK-NEXT: vfma.f32 q0, q5, r3
+; CHECK-NEXT: vfma.f32 q0, q5, r6
; CHECK-NEXT: ldr r0, [sp, #20] @ 4-byte Reload
; CHECK-NEXT: vfma.f32 q0, q2, lr
; CHECK-NEXT: vldrw.u32 q1, [r5, #-4]
@@ -1149,26 +1147,26 @@ define void @fir(%struct.arm_fir_instance_f32* nocapture readonly %S, float* noc
; CHECK-NEXT: .LBB16_8: @ %for.end
; CHECK-NEXT: @ in Loop: Header=BB16_4 Depth=1
; CHECK-NEXT: ldrd r9, r1, [sp, #24] @ 8-byte Folded Reload
-; CHECK-NEXT: ldr r3, [sp, #12] @ 4-byte Reload
-; CHECK-NEXT: cmp.w r3, #0
+; CHECK-NEXT: ldr r4, [sp, #12] @ 4-byte Reload
+; CHECK-NEXT: cmp.w r4, #0
; CHECK-NEXT: beq .LBB16_3
; CHECK-NEXT: b .LBB16_9
; CHECK-NEXT: .LBB16_9: @ %while.body76.preheader
; CHECK-NEXT: @ in Loop: Header=BB16_4 Depth=1
-; CHECK-NEXT: mov r6, r5
-; CHECK-NEXT: mov lr, r3
+; CHECK-NEXT: mov r3, r5
+; CHECK-NEXT: mov lr, r4
; CHECK-NEXT: .LBB16_10: @ %while.body76
; CHECK-NEXT: @ Parent Loop BB16_4 Depth=1
; CHECK-NEXT: @ => This Inner Loop Header: Depth=2
; CHECK-NEXT: ldr r0, [r7], #4
-; CHECK-NEXT: vldrw.u32 q1, [r6], #4
+; CHECK-NEXT: vldrw.u32 q1, [r3], #4
; CHECK-NEXT: subs.w lr, lr, #1
; CHECK-NEXT: vfma.f32 q0, q1, r0
; CHECK-NEXT: bne .LBB16_10
; CHECK-NEXT: b .LBB16_11
; CHECK-NEXT: .LBB16_11: @ %while.end.loopexit
; CHECK-NEXT: @ in Loop: Header=BB16_4 Depth=1
-; CHECK-NEXT: add.w r5, r5, r3, lsl #2
+; CHECK-NEXT: add.w r5, r5, r4, lsl #2
; CHECK-NEXT: b .LBB16_3
; CHECK-NEXT: .LBB16_12: @ %if.end
; CHECK-NEXT: add sp, #32
diff --git a/llvm/test/CodeGen/Thumb2/mve-postinc-lsr.ll b/llvm/test/CodeGen/Thumb2/mve-postinc-lsr.ll
index ea9ce9e51a7a..b297ea03ecee 100644
--- a/llvm/test/CodeGen/Thumb2/mve-postinc-lsr.ll
+++ b/llvm/test/CodeGen/Thumb2/mve-postinc-lsr.ll
@@ -213,14 +213,14 @@ define i8* @test(i8* nocapture readonly %input_row, i8* nocapture readonly %inpu
; CHECK-NEXT: cmp r3, #4
; CHECK-NEXT: strd r0, r1, [sp, #12] @ 8-byte Folded Spill
; CHECK-NEXT: bne .LBB2_8
-; CHECK-NEXT: @ %bb.1: @ %for.cond.preheader
+; CHECK-NEXT: @ %bb.1: @ %entry
; CHECK-NEXT: cmp r2, #0
; CHECK-NEXT: beq .LBB2_8
; CHECK-NEXT: @ %bb.2: @ %for.body.lr.ph
; CHECK-NEXT: ldr r3, [sp, #64]
; CHECK-NEXT: mov.w r9, #0
; CHECK-NEXT: ldr r1, [sp, #16] @ 4-byte Reload
-; CHECK-NEXT: ldr.w r11, [sp, #56]
+; CHECK-NEXT: ldr r4, [sp, #56]
; CHECK-NEXT: add.w r0, r1, r3, lsl #1
; CHECK-NEXT: str r0, [sp, #8] @ 4-byte Spill
; CHECK-NEXT: adds r0, r1, r3
@@ -229,57 +229,57 @@ define i8* @test(i8* nocapture readonly %input_row, i8* nocapture readonly %inpu
; CHECK-NEXT: add r0, r1
; CHECK-NEXT: str r0, [sp] @ 4-byte Spill
; CHECK-NEXT: adds r0, r3, #7
-; CHECK-NEXT: lsrs r0, r0, #3
+; CHECK-NEXT: lsr.w r11, r0, #3
; CHECK-NEXT: b .LBB2_5
; CHECK-NEXT: .LBB2_3: @ in Loop: Header=BB2_5 Depth=1
-; CHECK-NEXT: mov r10, r12
-; CHECK-NEXT: mov r8, r12
-; CHECK-NEXT: mov r6, r12
+; CHECK-NEXT: mov r12, r10
+; CHECK-NEXT: mov r8, r10
+; CHECK-NEXT: mov r6, r10
; CHECK-NEXT: .LBB2_4: @ %for.cond.cleanup23
; CHECK-NEXT: @ in Loop: Header=BB2_5 Depth=1
-; CHECK-NEXT: ldr r3, [sp, #72]
-; CHECK-NEXT: add.w r1, r8, r10
-; CHECK-NEXT: add r1, r6
-; CHECK-NEXT: add r1, r12
-; CHECK-NEXT: strb.w r1, [r3, r9]
+; CHECK-NEXT: ldr r1, [sp, #72]
+; CHECK-NEXT: add.w r0, r8, r12
+; CHECK-NEXT: add r0, r6
+; CHECK-NEXT: add r0, r10
+; CHECK-NEXT: strb.w r0, [r1, r9]
; CHECK-NEXT: add.w r9, r9, #1
; CHECK-NEXT: cmp r9, r2
; CHECK-NEXT: beq .LBB2_8
; CHECK-NEXT: .LBB2_5: @ %for.body
; CHECK-NEXT: @ =>This Loop Header: Depth=1
; CHECK-NEXT: @ Child Loop BB2_7 Depth 2
-; CHECK-NEXT: ldr r1, [sp, #68]
-; CHECK-NEXT: ldr.w r12, [r1, r9, lsl #2]
-; CHECK-NEXT: subs r1, r0, r0
+; CHECK-NEXT: ldr r0, [sp, #68]
+; CHECK-NEXT: ldr.w r10, [r0, r9, lsl #2]
+; CHECK-NEXT: subs.w r0, r11, r11
; CHECK-NEXT: ble .LBB2_3
; CHECK-NEXT: @ %bb.6: @ %for.body24.preheader
; CHECK-NEXT: @ in Loop: Header=BB2_5 Depth=1
-; CHECK-NEXT: ldr r7, [sp, #64]
-; CHECK-NEXT: mov r6, r12
-; CHECK-NEXT: ldr r3, [sp, #12] @ 4-byte Reload
-; CHECK-NEXT: dls lr, r1
+; CHECK-NEXT: ldr r3, [sp, #64]
+; CHECK-NEXT: mov r6, r10
+; CHECK-NEXT: ldr r1, [sp, #12] @ 4-byte Reload
+; CHECK-NEXT: dls lr, r0
+; CHECK-NEXT: ldrd r5, r0, [sp] @ 8-byte Folded Reload
+; CHECK-NEXT: mov r8, r10
+; CHECK-NEXT: mla r7, r9, r3, r1
; CHECK-NEXT: ldr r1, [sp, #16] @ 4-byte Reload
-; CHECK-NEXT: mov r8, r12
-; CHECK-NEXT: mla r7, r9, r7, r3
-; CHECK-NEXT: ldr r5, [sp, #8] @ 4-byte Reload
-; CHECK-NEXT: ldrd r4, r3, [sp] @ 8-byte Folded Reload
-; CHECK-NEXT: mov r10, r12
+; CHECK-NEXT: ldr r3, [sp, #8] @ 4-byte Reload
+; CHECK-NEXT: mov r12, r10
; CHECK-NEXT: .LBB2_7: @ %for.body24
; CHECK-NEXT: @ Parent Loop BB2_5 Depth=1
; CHECK-NEXT: @ => This Inner Loop Header: Depth=2
-; CHECK-NEXT: vldrb.s16 q0, [r4], #8
-; CHECK-NEXT: vadd.i16 q1, q0, r11
+; CHECK-NEXT: vldrb.s16 q0, [r5], #8
+; CHECK-NEXT: vadd.i16 q1, q0, r4
; CHECK-NEXT: vldrb.s16 q0, [r7], #8
-; CHECK-NEXT: vmlava.s16 r12, q0, q1
-; CHECK-NEXT: vldrb.s16 q1, [r5], #8
-; CHECK-NEXT: vadd.i16 q1, q1, r11
-; CHECK-NEXT: vmlava.s16 r6, q0, q1
+; CHECK-NEXT: vmlava.s16 r10, q0, q1
; CHECK-NEXT: vldrb.s16 q1, [r3], #8
-; CHECK-NEXT: vadd.i16 q1, q1, r11
+; CHECK-NEXT: vadd.i16 q1, q1, r4
+; CHECK-NEXT: vmlava.s16 r6, q0, q1
+; CHECK-NEXT: vldrb.s16 q1, [r0], #8
+; CHECK-NEXT: vadd.i16 q1, q1, r4
; CHECK-NEXT: vmlava.s16 r8, q0, q1
; CHECK-NEXT: vldrb.s16 q1, [r1], #8
-; CHECK-NEXT: vadd.i16 q1, q1, r11
-; CHECK-NEXT: vmlava.s16 r10, q0, q1
+; CHECK-NEXT: vadd.i16 q1, q1, r4
+; CHECK-NEXT: vmlava.s16 r12, q0, q1
; CHECK-NEXT: le lr, .LBB2_7
; CHECK-NEXT: b .LBB2_4
; CHECK-NEXT: .LBB2_8: @ %if.end
@@ -396,14 +396,14 @@ define i8* @test_optsize(i8* nocapture readonly %input_row, i8* nocapture readon
; CHECK-NEXT: cmp r3, #4
; CHECK-NEXT: strd r0, r1, [sp, #12] @ 8-byte Folded Spill
; CHECK-NEXT: bne .LBB3_8
-; CHECK-NEXT: @ %bb.1: @ %for.cond.preheader
+; CHECK-NEXT: @ %bb.1: @ %entry
; CHECK-NEXT: cmp r2, #0
; CHECK-NEXT: beq .LBB3_8
; CHECK-NEXT: @ %bb.2: @ %for.body.lr.ph
; CHECK-NEXT: ldr r3, [sp, #64]
; CHECK-NEXT: mov.w r9, #0
; CHECK-NEXT: ldr r1, [sp, #16] @ 4-byte Reload
-; CHECK-NEXT: ldr.w r11, [sp, #56]
+; CHECK-NEXT: ldr r4, [sp, #56]
; CHECK-NEXT: add.w r0, r1, r3, lsl #1
; CHECK-NEXT: str r0, [sp, #8] @ 4-byte Spill
; CHECK-NEXT: adds r0, r1, r3
@@ -412,55 +412,55 @@ define i8* @test_optsize(i8* nocapture readonly %input_row, i8* nocapture readon
; CHECK-NEXT: add r0, r1
; CHECK-NEXT: str r0, [sp] @ 4-byte Spill
; CHECK-NEXT: adds r0, r3, #7
-; CHECK-NEXT: lsrs r0, r0, #3
+; CHECK-NEXT: lsr.w r11, r0, #3
; CHECK-NEXT: .LBB3_3: @ %for.body
; CHECK-NEXT: @ =>This Loop Header: Depth=1
; CHECK-NEXT: @ Child Loop BB3_5 Depth 2
-; CHECK-NEXT: ldr r1, [sp, #68]
-; CHECK-NEXT: ldr.w r12, [r1, r9, lsl #2]
-; CHECK-NEXT: subs r1, r0, r0
+; CHECK-NEXT: ldr r0, [sp, #68]
+; CHECK-NEXT: ldr.w r10, [r0, r9, lsl #2]
+; CHECK-NEXT: subs.w r0, r11, r11
; CHECK-NEXT: ble .LBB3_6
; CHECK-NEXT: @ %bb.4: @ %for.body24.preheader
; CHECK-NEXT: @ in Loop: Header=BB3_3 Depth=1
-; CHECK-NEXT: ldr r7, [sp, #64]
-; CHECK-NEXT: mov r6, r12
-; CHECK-NEXT: ldr r3, [sp, #12] @ 4-byte Reload
-; CHECK-NEXT: dls lr, r1
+; CHECK-NEXT: ldr r3, [sp, #64]
+; CHECK-NEXT: mov r6, r10
+; CHECK-NEXT: ldr r1, [sp, #12] @ 4-byte Reload
+; CHECK-NEXT: dls lr, r0
+; CHECK-NEXT: ldrd r5, r0, [sp] @ 8-byte Folded Reload
+; CHECK-NEXT: mov r8, r10
+; CHECK-NEXT: mla r7, r9, r3, r1
; CHECK-NEXT: ldr r1, [sp, #16] @ 4-byte Reload
-; CHECK-NEXT: mov r8, r12
-; CHECK-NEXT: mla r7, r9, r7, r3
-; CHECK-NEXT: ldr r5, [sp, #8] @ 4-byte Reload
-; CHECK-NEXT: ldrd r4, r3, [sp] @ 8-byte Folded Reload
-; CHECK-NEXT: mov r10, r12
+; CHECK-NEXT: ldr r3, [sp, #8] @ 4-byte Reload
+; CHECK-NEXT: mov r12, r10
; CHECK-NEXT: .LBB3_5: @ %for.body24
; CHECK-NEXT: @ Parent Loop BB3_3 Depth=1
; CHECK-NEXT: @ => This Inner Loop Header: Depth=2
-; CHECK-NEXT: vldrb.s16 q0, [r4], #8
-; CHECK-NEXT: vadd.i16 q1, q0, r11
+; CHECK-NEXT: vldrb.s16 q0, [r5], #8
+; CHECK-NEXT: vadd.i16 q1, q0, r4
; CHECK-NEXT: vldrb.s16 q0, [r7], #8
-; CHECK-NEXT: vmlava.s16 r12, q0, q1
-; CHECK-NEXT: vldrb.s16 q1, [r5], #8
-; CHECK-NEXT: vadd.i16 q1, q1, r11
-; CHECK-NEXT: vmlava.s16 r6, q0, q1
+; CHECK-NEXT: vmlava.s16 r10, q0, q1
; CHECK-NEXT: vldrb.s16 q1, [r3], #8
-; CHECK-NEXT: vadd.i16 q1, q1, r11
+; CHECK-NEXT: vadd.i16 q1, q1, r4
+; CHECK-NEXT: vmlava.s16 r6, q0, q1
+; CHECK-NEXT: vldrb.s16 q1, [r0], #8
+; CHECK-NEXT: vadd.i16 q1, q1, r4
; CHECK-NEXT: vmlava.s16 r8, q0, q1
; CHECK-NEXT: vldrb.s16 q1, [r1], #8
-; CHECK-NEXT: vadd.i16 q1, q1, r11
-; CHECK-NEXT: vmlava.s16 r10, q0, q1
+; CHECK-NEXT: vadd.i16 q1, q1, r4
+; CHECK-NEXT: vmlava.s16 r12, q0, q1
; CHECK-NEXT: le lr, .LBB3_5
; CHECK-NEXT: b .LBB3_7
; CHECK-NEXT: .LBB3_6: @ in Loop: Header=BB3_3 Depth=1
-; CHECK-NEXT: mov r10, r12
-; CHECK-NEXT: mov r8, r12
-; CHECK-NEXT: mov r6, r12
+; CHECK-NEXT: mov r12, r10
+; CHECK-NEXT: mov r8, r10
+; CHECK-NEXT: mov r6, r10
; CHECK-NEXT: .LBB3_7: @ %for.cond.cleanup23
; CHECK-NEXT: @ in Loop: Header=BB3_3 Depth=1
-; CHECK-NEXT: ldr r3, [sp, #72]
-; CHECK-NEXT: add.w r1, r8, r10
-; CHECK-NEXT: add r1, r6
-; CHECK-NEXT: add r1, r12
-; CHECK-NEXT: strb.w r1, [r3, r9]
+; CHECK-NEXT: ldr r1, [sp, #72]
+; CHECK-NEXT: add.w r0, r8, r12
+; CHECK-NEXT: add r0, r6
+; CHECK-NEXT: add r0, r10
+; CHECK-NEXT: strb.w r0, [r1, r9]
; CHECK-NEXT: add.w r9, r9, #1
; CHECK-NEXT: cmp r9, r2
; CHECK-NEXT: bne .LBB3_3
@@ -689,7 +689,7 @@ define i8* @signext(i8* %input_row, i8* %input_col, i16 zeroext %output_ch, i16
; CHECK-NEXT: cmp r3, #4
; CHECK-NEXT: stm.w r12, {r0, r1, r2} @ 12-byte Folded Spill
; CHECK-NEXT: bne .LBB5_8
-; CHECK-NEXT: @ %bb.1: @ %for.cond.preheader
+; CHECK-NEXT: @ %bb.1: @ %entry
; CHECK-NEXT: ldr r0, [sp, #20] @ 4-byte Reload
; CHECK-NEXT: cmp r0, #0
; CHECK-NEXT: beq .LBB5_8
@@ -697,7 +697,7 @@ define i8* @signext(i8* %input_row, i8* %input_col, i16 zeroext %output_ch, i16
; CHECK-NEXT: ldr r2, [sp, #88]
; CHECK-NEXT: mov.w r9, #0
; CHECK-NEXT: ldr r1, [sp, #16] @ 4-byte Reload
-; CHECK-NEXT: ldr r4, [sp, #72]
+; CHECK-NEXT: ldr.w r10, [sp, #72]
; CHECK-NEXT: add.w r0, r1, r2, lsl #1
; CHECK-NEXT: str r0, [sp, #8] @ 4-byte Spill
; CHECK-NEXT: adds r0, r1, r2
@@ -709,15 +709,15 @@ define i8* @signext(i8* %input_row, i8* %input_col, i16 zeroext %output_ch, i16
; CHECK-NEXT: lsrs r2, r0, #3
; CHECK-NEXT: b .LBB5_5
; CHECK-NEXT: .LBB5_3: @ in Loop: Header=BB5_5 Depth=1
-; CHECK-NEXT: mov r10, r12
-; CHECK-NEXT: mov r8, r12
-; CHECK-NEXT: mov r6, r12
+; CHECK-NEXT: mov r8, r0
+; CHECK-NEXT: mov r12, r0
+; CHECK-NEXT: mov r6, r0
; CHECK-NEXT: .LBB5_4: @ %for.cond.cleanup23
; CHECK-NEXT: @ in Loop: Header=BB5_5 Depth=1
-; CHECK-NEXT: add.w r0, r8, r10
+; CHECK-NEXT: add.w r1, r12, r8
+; CHECK-NEXT: add r1, r6
+; CHECK-NEXT: add r0, r1
; CHECK-NEXT: ldr r1, [sp, #96]
-; CHECK-NEXT: add r0, r6
-; CHECK-NEXT: add r0, r12
; CHECK-NEXT: strb.w r0, [r1, r9]
; CHECK-NEXT: add.w r9, r9, #1
; CHECK-NEXT: ldr r0, [sp, #20] @ 4-byte Reload
@@ -728,36 +728,35 @@ define i8* @signext(i8* %input_row, i8* %input_col, i16 zeroext %output_ch, i16
; CHECK-NEXT: @ Child Loop BB5_7 Depth 2
; CHECK-NEXT: ldr r0, [sp, #92]
; CHECK-NEXT: cmp r2, r2
-; CHECK-NEXT: ldr.w r12, [r0, r9, lsl #2]
+; CHECK-NEXT: ldr.w r0, [r0, r9, lsl #2]
; CHECK-NEXT: bge .LBB5_3
; CHECK-NEXT: @ %bb.6: @ %for.body24.preheader
; CHECK-NEXT: @ in Loop: Header=BB5_5 Depth=1
; CHECK-NEXT: ldr.w r11, [sp, #88]
-; CHECK-NEXT: ldr r0, [sp, #12] @ 4-byte Reload
-; CHECK-NEXT: mov r6, r12
+; CHECK-NEXT: ldr r1, [sp, #12] @ 4-byte Reload
+; CHECK-NEXT: mov r6, r0
; CHECK-NEXT: dlstp.16 lr, r11
+; CHECK-NEXT: ldm.w sp, {r4, r5, r7} @ 12-byte Folded Reload
+; CHECK-NEXT: mov r12, r0
+; CHECK-NEXT: mla r3, r9, r11, r1
; CHECK-NEXT: ldr r1, [sp, #16] @ 4-byte Reload
-; CHECK-NEXT: mla r3, r9, r11, r0
-; CHECK-NEXT: ldr r0, [sp, #8] @ 4-byte Reload
-; CHECK-NEXT: ldrd r7, r5, [sp] @ 8-byte Folded Reload
-; CHECK-NEXT: mov r8, r12
-; CHECK-NEXT: mov r10, r12
+; CHECK-NEXT: mov r8, r0
; CHECK-NEXT: .LBB5_7: @ %for.body24
; CHECK-NEXT: @ Parent Loop BB5_5 Depth=1
; CHECK-NEXT: @ => This Inner Loop Header: Depth=2
-; CHECK-NEXT: vldrb.s16 q0, [r7], #8
-; CHECK-NEXT: vadd.i16 q1, q0, r4
+; CHECK-NEXT: vldrb.s16 q0, [r4], #8
+; CHECK-NEXT: vadd.i16 q1, q0, r10
; CHECK-NEXT: vldrb.s16 q0, [r3], #8
-; CHECK-NEXT: vmlava.s16 r12, q0, q1
-; CHECK-NEXT: vldrb.s16 q1, [r0], #8
-; CHECK-NEXT: vadd.i16 q1, q1, r4
+; CHECK-NEXT: vmlava.s16 r0, q0, q1
+; CHECK-NEXT: vldrb.s16 q1, [r7], #8
+; CHECK-NEXT: vadd.i16 q1, q1, r10
; CHECK-NEXT: vmlava.s16 r6, q0, q1
; CHECK-NEXT: vldrb.s16 q1, [r5], #8
-; CHECK-NEXT: vadd.i16 q1, q1, r4
-; CHECK-NEXT: vmlava.s16 r8, q0, q1
+; CHECK-NEXT: vadd.i16 q1, q1, r10
+; CHECK-NEXT: vmlava.s16 r12, q0, q1
; CHECK-NEXT: vldrb.s16 q1, [r1], #8
-; CHECK-NEXT: vadd.i16 q1, q1, r4
-; CHECK-NEXT: vmlava.s16 r10, q0, q1
+; CHECK-NEXT: vadd.i16 q1, q1, r10
+; CHECK-NEXT: vmlava.s16 r8, q0, q1
; CHECK-NEXT: letp lr, .LBB5_7
; CHECK-NEXT: b .LBB5_4
; CHECK-NEXT: .LBB5_8: @ %if.end
@@ -877,7 +876,7 @@ define i8* @signext_optsize(i8* %input_row, i8* %input_col, i16 zeroext %output_
; CHECK-NEXT: cmp r3, #4
; CHECK-NEXT: stm.w r12, {r0, r1, r2} @ 12-byte Folded Spill
; CHECK-NEXT: bne .LBB6_8
-; CHECK-NEXT: @ %bb.1: @ %for.cond.preheader
+; CHECK-NEXT: @ %bb.1: @ %entry
; CHECK-NEXT: ldr r0, [sp, #20] @ 4-byte Reload
; CHECK-NEXT: cmp r0, #0
; CHECK-NEXT: beq .LBB6_8
@@ -885,7 +884,7 @@ define i8* @signext_optsize(i8* %input_row, i8* %input_col, i16 zeroext %output_
; CHECK-NEXT: ldr r2, [sp, #88]
; CHECK-NEXT: mov.w r9, #0
; CHECK-NEXT: ldr r1, [sp, #16] @ 4-byte Reload
-; CHECK-NEXT: ldr r4, [sp, #72]
+; CHECK-NEXT: ldr.w r10, [sp, #72]
; CHECK-NEXT: add.w r0, r1, r2, lsl #1
; CHECK-NEXT: str r0, [sp, #8] @ 4-byte Spill
; CHECK-NEXT: adds r0, r1, r2
@@ -900,48 +899,47 @@ define i8* @signext_optsize(i8* %input_row, i8* %input_col, i16 zeroext %output_
; CHECK-NEXT: @ Child Loop BB6_5 Depth 2
; CHECK-NEXT: ldr r0, [sp, #92]
; CHECK-NEXT: cmp r2, r2
-; CHECK-NEXT: ldr.w r12, [r0, r9, lsl #2]
+; CHECK-NEXT: ldr.w r0, [r0, r9, lsl #2]
; CHECK-NEXT: bge .LBB6_6
; CHECK-NEXT: @ %bb.4: @ %for.body24.preheader
; CHECK-NEXT: @ in Loop: Header=BB6_3 Depth=1
; CHECK-NEXT: ldr.w r11, [sp, #88]
-; CHECK-NEXT: ldr r0, [sp, #12] @ 4-byte Reload
-; CHECK-NEXT: mov r6, r12
+; CHECK-NEXT: ldr r1, [sp, #12] @ 4-byte Reload
+; CHECK-NEXT: mov r6, r0
; CHECK-NEXT: dlstp.16 lr, r11
+; CHECK-NEXT: ldm.w sp, {r4, r5, r7} @ 12-byte Folded Reload
+; CHECK-NEXT: mov r12, r0
+; CHECK-NEXT: mla r3, r9, r11, r1
; CHECK-NEXT: ldr r1, [sp, #16] @ 4-byte Reload
-; CHECK-NEXT: mla r3, r9, r11, r0
-; CHECK-NEXT: ldr r0, [sp, #8] @ 4-byte Reload
-; CHECK-NEXT: ldrd r7, r5, [sp] @ 8-byte Folded Reload
-; CHECK-NEXT: mov r8, r12
-; CHECK-NEXT: mov r10, r12
+; CHECK-NEXT: mov r8, r0
; CHECK-NEXT: .LBB6_5: @ %for.body24
; CHECK-NEXT: @ Parent Loop BB6_3 Depth=1
; CHECK-NEXT: @ => This Inner Loop Header: Depth=2
-; CHECK-NEXT: vldrb.s16 q0, [r7], #8
-; CHECK-NEXT: vadd.i16 q1, q0, r4
+; CHECK-NEXT: vldrb.s16 q0, [r4], #8
+; CHECK-NEXT: vadd.i16 q1, q0, r10
; CHECK-NEXT: vldrb.s16 q0, [r3], #8
-; CHECK-NEXT: vmlava.s16 r12, q0, q1
-; CHECK-NEXT: vldrb.s16 q1, [r0], #8
-; CHECK-NEXT: vadd.i16 q1, q1, r4
+; CHECK-NEXT: vmlava.s16 r0, q0, q1
+; CHECK-NEXT: vldrb.s16 q1, [r7], #8
+; CHECK-NEXT: vadd.i16 q1, q1, r10
; CHECK-NEXT: vmlava.s16 r6, q0, q1
; CHECK-NEXT: vldrb.s16 q1, [r5], #8
-; CHECK-NEXT: vadd.i16 q1, q1, r4
-; CHECK-NEXT: vmlava.s16 r8, q0, q1
+; CHECK-NEXT: vadd.i16 q1, q1, r10
+; CHECK-NEXT: vmlava.s16 r12, q0, q1
; CHECK-NEXT: vldrb.s16 q1, [r1], #8
-; CHECK-NEXT: vadd.i16 q1, q1, r4
-; CHECK-NEXT: vmlava.s16 r10, q0, q1
+; CHECK-NEXT: vadd.i16 q1, q1, r10
+; CHECK-NEXT: vmlava.s16 r8, q0, q1
; CHECK-NEXT: letp lr, .LBB6_5
; CHECK-NEXT: b .LBB6_7
; CHECK-NEXT: .LBB6_6: @ in Loop: Header=BB6_3 Depth=1
-; CHECK-NEXT: mov r10, r12
-; CHECK-NEXT: mov r8, r12
-; CHECK-NEXT: mov r6, r12
+; CHECK-NEXT: mov r8, r0
+; CHECK-NEXT: mov r12, r0
+; CHECK-NEXT: mov r6, r0
; CHECK-NEXT: .LBB6_7: @ %for.cond.cleanup23
; CHECK-NEXT: @ in Loop: Header=BB6_3 Depth=1
-; CHECK-NEXT: add.w r0, r8, r10
+; CHECK-NEXT: add.w r1, r12, r8
+; CHECK-NEXT: add r1, r6
+; CHECK-NEXT: add r0, r1
; CHECK-NEXT: ldr r1, [sp, #96]
-; CHECK-NEXT: add r0, r6
-; CHECK-NEXT: add r0, r12
; CHECK-NEXT: strb.w r0, [r1, r9]
; CHECK-NEXT: add.w r9, r9, #1
; CHECK-NEXT: ldr r0, [sp, #20] @ 4-byte Reload
diff --git a/llvm/test/Transforms/LoopUnroll/peel-loop-inner.ll b/llvm/test/Transforms/LoopUnroll/peel-loop-inner.ll
index 7b088b31a54c..8d89053c96f1 100644
--- a/llvm/test/Transforms/LoopUnroll/peel-loop-inner.ll
+++ b/llvm/test/Transforms/LoopUnroll/peel-loop-inner.ll
@@ -1,3 +1,4 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
; RUN: opt < %s -S -passes='require<opt-remark-emit>,loop-unroll<peeling;no-runtime>,simplify-cfg,instcombine' -unroll-force-peel-count=3 -verify-dom-info | FileCheck %s
define void @basic(i32 %K, i32 %N) {
@@ -6,24 +7,19 @@ define void @basic(i32 %K, i32 %N) {
; CHECK-NEXT: br label [[OUTER:%.*]]
; CHECK: outer:
; CHECK-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[I_INC:%.*]], [[OUTER_BACKEDGE:%.*]] ]
-; CHECK-NEXT: [[CMP_INNER_PEEL:%.*]] = icmp sgt i32 [[K:%.*]], 1
-; CHECK-NEXT: br i1 [[CMP_INNER_PEEL]], label [[INNER_PEEL2:%.*]], label [[OUTER_BACKEDGE]]
-; CHECK: inner.peel2:
-; CHECK-NEXT: [[CMP_INNER_PEEL4:%.*]] = icmp eq i32 [[K]], 2
-; CHECK-NEXT: br i1 [[CMP_INNER_PEEL4]], label [[OUTER_BACKEDGE]], label [[INNER_PEEL6:%.*]]
-; CHECK: inner.peel6:
-; CHECK-NEXT: [[CMP_INNER_PEEL8:%.*]] = icmp sgt i32 [[K]], 3
+; CHECK-NEXT: [[CMP_INNER_PEEL8:%.*]] = icmp sgt i32 [[K:%.*]], 3
; CHECK-NEXT: br i1 [[CMP_INNER_PEEL8]], label [[INNER:%.*]], label [[OUTER_BACKEDGE]]
; CHECK: inner:
-; CHECK-NEXT: [[J:%.*]] = phi i32 [ [[J_INC:%.*]], [[INNER]] ], [ 3, [[INNER_PEEL6]] ]
+; CHECK-NEXT: [[J:%.*]] = phi i32 [ [[J_INC:%.*]], [[INNER]] ], [ 3, [[OUTER]] ]
; CHECK-NEXT: [[J_INC]] = add nuw nsw i32 [[J]], 1
; CHECK-NEXT: [[CMP_INNER:%.*]] = icmp slt i32 [[J_INC]], [[K]]
-; CHECK-NEXT: br i1 [[CMP_INNER]], label [[INNER]], label [[OUTER_BACKEDGE]], !llvm.loop !0
+; CHECK-NEXT: br i1 [[CMP_INNER]], label [[INNER]], label [[OUTER_BACKEDGE]], [[LOOP0:!llvm.loop !.*]]
; CHECK: outer.backedge:
; CHECK-NEXT: [[I_INC]] = add i32 [[I]], 1
; CHECK-NEXT: [[CMP_OUTER:%.*]] = icmp slt i32 [[I_INC]], [[N:%.*]]
-; CHECK-NOT: !llvm.loop
+; CHECK-NEXT: br i1 [[CMP_OUTER]], label [[OUTER]], label [[END:%.*]]
; CHECK: end:
+; CHECK-NEXT: ret void
;
entry:
br label %outer
diff --git a/llvm/test/Transforms/SimplifyCFG/fold-branch-to-common-dest.ll b/llvm/test/Transforms/SimplifyCFG/fold-branch-to-common-dest.ll
index 6f83a20c72da..22f2ed147b64 100644
--- a/llvm/test/Transforms/SimplifyCFG/fold-branch-to-common-dest.ll
+++ b/llvm/test/Transforms/SimplifyCFG/fold-branch-to-common-dest.ll
@@ -161,12 +161,11 @@ define void @one_pred_with_extra_op_multiuse(i8 %v0, i8 %v1) {
; CHECK-LABEL: @one_pred_with_extra_op_multiuse(
; CHECK-NEXT: pred:
; CHECK-NEXT: [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
-; CHECK-NEXT: br i1 [[C0]], label [[DISPATCH:%.*]], label [[FINAL_RIGHT:%.*]]
-; CHECK: dispatch:
; CHECK-NEXT: [[V1_ADJ:%.*]] = add i8 [[V0]], [[V1:%.*]]
; CHECK-NEXT: [[V1_ADJ_ADJ:%.*]] = add i8 [[V1_ADJ]], [[V1_ADJ]]
; CHECK-NEXT: [[C1:%.*]] = icmp eq i8 [[V1_ADJ_ADJ]], 0
-; CHECK-NEXT: br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT]]
+; CHECK-NEXT: [[OR_COND:%.*]] = and i1 [[C0]], [[C1]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT:%.*]]
; CHECK: final_left:
; CHECK-NEXT: call void @sideeffect0()
; CHECK-NEXT: ret void
@@ -197,15 +196,18 @@ define void @two_preds_with_extra_op_multiuse(i8 %v0, i8 %v1, i8 %v2, i8 %v3) {
; CHECK-NEXT: br i1 [[C0]], label [[PRED0:%.*]], label [[PRED1:%.*]]
; CHECK: pred0:
; CHECK-NEXT: [[C1:%.*]] = icmp eq i8 [[V1:%.*]], 0
-; CHECK-NEXT: br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[DISPATCH:%.*]]
+; CHECK-NEXT: [[DOTOLD:%.*]] = add i8 [[V1]], [[V2:%.*]]
+; CHECK-NEXT: [[DOTOLD1:%.*]] = add i8 [[DOTOLD]], [[DOTOLD]]
+; CHECK-NEXT: [[C3_OLD:%.*]] = icmp eq i8 [[DOTOLD1]], 0
+; CHECK-NEXT: [[OR_COND4:%.*]] = or i1 [[C1]], [[C3_OLD]]
+; CHECK-NEXT: br i1 [[OR_COND4]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT:%.*]]
; CHECK: pred1:
-; CHECK-NEXT: [[C2:%.*]] = icmp eq i8 [[V2:%.*]], 0
-; CHECK-NEXT: br i1 [[C2]], label [[DISPATCH]], label [[FINAL_RIGHT:%.*]]
-; CHECK: dispatch:
+; CHECK-NEXT: [[C2:%.*]] = icmp eq i8 [[V2]], 0
; CHECK-NEXT: [[V3_ADJ:%.*]] = add i8 [[V1]], [[V2]]
; CHECK-NEXT: [[V3_ADJ_ADJ:%.*]] = add i8 [[V3_ADJ]], [[V3_ADJ]]
; CHECK-NEXT: [[C3:%.*]] = icmp eq i8 [[V3_ADJ_ADJ]], 0
-; CHECK-NEXT: br i1 [[C3]], label [[FINAL_LEFT]], label [[FINAL_RIGHT]]
+; CHECK-NEXT: [[OR_COND:%.*]] = and i1 [[C2]], [[C3]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[FINAL_LEFT]], label [[FINAL_RIGHT]]
; CHECK: final_left:
; CHECK-NEXT: call void @sideeffect0()
; CHECK-NEXT: ret void
@@ -242,11 +244,10 @@ define void @one_pred_with_extra_op_liveout(i8 %v0, i8 %v1) {
; CHECK-LABEL: @one_pred_with_extra_op_liveout(
; CHECK-NEXT: pred:
; CHECK-NEXT: [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
-; CHECK-NEXT: br i1 [[C0]], label [[DISPATCH:%.*]], label [[FINAL_RIGHT:%.*]]
-; CHECK: dispatch:
; CHECK-NEXT: [[V1_ADJ:%.*]] = add i8 [[V0]], [[V1:%.*]]
; CHECK-NEXT: [[C1:%.*]] = icmp eq i8 [[V1_ADJ]], 0
-; CHECK-NEXT: br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT]]
+; CHECK-NEXT: [[OR_COND:%.*]] = and i1 [[C0]], [[C1]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT:%.*]]
; CHECK: final_left:
; CHECK-NEXT: call void @sideeffect0()
; CHECK-NEXT: call void @use8(i8 [[V1_ADJ]])
@@ -274,11 +275,10 @@ define void @one_pred_with_extra_op_liveout_multiuse(i8 %v0, i8 %v1) {
; CHECK-LABEL: @one_pred_with_extra_op_liveout_multiuse(
; CHECK-NEXT: pred:
; CHECK-NEXT: [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
-; CHECK-NEXT: br i1 [[C0]], label [[DISPATCH:%.*]], label [[FINAL_RIGHT:%.*]]
-; CHECK: dispatch:
; CHECK-NEXT: [[V1_ADJ:%.*]] = add i8 [[V0]], [[V1:%.*]]
; CHECK-NEXT: [[C1:%.*]] = icmp eq i8 [[V1_ADJ]], 0
-; CHECK-NEXT: br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT]]
+; CHECK-NEXT: [[OR_COND:%.*]] = and i1 [[C0]], [[C1]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT:%.*]]
; CHECK: final_left:
; CHECK-NEXT: call void @sideeffect0()
; CHECK-NEXT: call void @use8(i8 [[V1_ADJ]])
@@ -312,11 +312,10 @@ define void @one_pred_with_extra_op_liveout_distant_phi(i8 %v0, i8 %v1) {
; CHECK-NEXT: br i1 [[C0]], label [[PRED:%.*]], label [[LEFT_END:%.*]]
; CHECK: pred:
; CHECK-NEXT: [[C1:%.*]] = icmp eq i8 [[V1:%.*]], 0
-; CHECK-NEXT: br i1 [[C1]], label [[DISPATCH:%.*]], label [[FINAL_RIGHT:%.*]]
-; CHECK: dispatch:
; CHECK-NEXT: [[V2_ADJ:%.*]] = add i8 [[V0]], [[V1]]
; CHECK-NEXT: [[C2:%.*]] = icmp eq i8 [[V2_ADJ]], 0
-; CHECK-NEXT: br i1 [[C2]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT]]
+; CHECK-NEXT: [[OR_COND:%.*]] = and i1 [[C1]], [[C2]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT:%.*]]
; CHECK: final_left:
; CHECK-NEXT: call void @sideeffect0()
; CHECK-NEXT: call void @use8(i8 [[V2_ADJ]])
@@ -364,13 +363,16 @@ define void @two_preds_with_extra_op_liveout(i8 %v0, i8 %v1, i8 %v2, i8 %v3) {
; CHECK-NEXT: br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[DISPATCH:%.*]]
; CHECK: pred1:
; CHECK-NEXT: [[C2:%.*]] = icmp eq i8 [[V2:%.*]], 0
-; CHECK-NEXT: br i1 [[C2]], label [[DISPATCH]], label [[FINAL_RIGHT:%.*]]
-; CHECK: dispatch:
; CHECK-NEXT: [[V3_ADJ:%.*]] = add i8 [[V1]], [[V2]]
; CHECK-NEXT: [[C3:%.*]] = icmp eq i8 [[V3_ADJ]], 0
-; CHECK-NEXT: br i1 [[C3]], label [[FINAL_LEFT]], label [[FINAL_RIGHT]]
+; CHECK-NEXT: [[OR_COND:%.*]] = and i1 [[C2]], [[C3]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[FINAL_LEFT]], label [[FINAL_RIGHT:%.*]]
+; CHECK: dispatch:
+; CHECK-NEXT: [[DOTOLD:%.*]] = add i8 [[V1]], [[V2]]
+; CHECK-NEXT: [[C3_OLD:%.*]] = icmp eq i8 [[DOTOLD]], 0
+; CHECK-NEXT: br i1 [[C3_OLD]], label [[FINAL_LEFT]], label [[FINAL_RIGHT]]
; CHECK: final_left:
-; CHECK-NEXT: [[MERGE_LEFT:%.*]] = phi i8 [ [[V3_ADJ]], [[DISPATCH]] ], [ 0, [[PRED0]] ]
+; CHECK-NEXT: [[MERGE_LEFT:%.*]] = phi i8 [ [[DOTOLD]], [[DISPATCH]] ], [ 0, [[PRED0]] ], [ [[V3_ADJ]], [[PRED1]] ]
; CHECK-NEXT: call void @use8(i8 [[MERGE_LEFT]])
; CHECK-NEXT: call void @sideeffect0()
; CHECK-NEXT: ret void
@@ -411,14 +413,17 @@ define void @two_preds_with_extra_op_liveout_multiuse(i8 %v0, i8 %v1, i8 %v2, i8
; CHECK-NEXT: br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[DISPATCH:%.*]]
; CHECK: pred1:
; CHECK-NEXT: [[C2:%.*]] = icmp eq i8 [[V2:%.*]], 0
-; CHECK-NEXT: br i1 [[C2]], label [[DISPATCH]], label [[FINAL_RIGHT:%.*]]
-; CHECK: dispatch:
; CHECK-NEXT: [[V3_ADJ:%.*]] = add i8 [[V1]], [[V2]]
; CHECK-NEXT: [[C3:%.*]] = icmp eq i8 [[V3_ADJ]], 0
-; CHECK-NEXT: br i1 [[C3]], label [[FINAL_LEFT]], label [[FINAL_RIGHT]]
+; CHECK-NEXT: [[OR_COND:%.*]] = and i1 [[C2]], [[C3]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[FINAL_LEFT]], label [[FINAL_RIGHT:%.*]]
+; CHECK: dispatch:
+; CHECK-NEXT: [[DOTOLD:%.*]] = add i8 [[V1]], [[V2]]
+; CHECK-NEXT: [[C3_OLD:%.*]] = icmp eq i8 [[DOTOLD]], 0
+; CHECK-NEXT: br i1 [[C3_OLD]], label [[FINAL_LEFT]], label [[FINAL_RIGHT]]
; CHECK: final_left:
-; CHECK-NEXT: [[MERGE_LEFT:%.*]] = phi i8 [ [[V3_ADJ]], [[DISPATCH]] ], [ 0, [[PRED0]] ]
-; CHECK-NEXT: [[MERGE_LEFT_2:%.*]] = phi i8 [ [[V3_ADJ]], [[DISPATCH]] ], [ 42, [[PRED0]] ]
+; CHECK-NEXT: [[MERGE_LEFT:%.*]] = phi i8 [ [[DOTOLD]], [[DISPATCH]] ], [ 0, [[PRED0]] ], [ [[V3_ADJ]], [[PRED1]] ]
+; CHECK-NEXT: [[MERGE_LEFT_2:%.*]] = phi i8 [ [[DOTOLD]], [[DISPATCH]] ], [ 42, [[PRED0]] ], [ [[V3_ADJ]], [[PRED1]] ]
; CHECK-NEXT: call void @use8(i8 [[MERGE_LEFT]])
; CHECK-NEXT: call void @use8(i8 [[MERGE_LEFT_2]])
; CHECK-NEXT: call void @sideeffect0()
@@ -459,11 +464,10 @@ define void @one_pred_with_extra_op_eexternally_used_only(i8 %v0, i8 %v1) {
; CHECK-LABEL: @one_pred_with_extra_op_eexternally_used_only(
; CHECK-NEXT: pred:
; CHECK-NEXT: [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
-; CHECK-NEXT: br i1 [[C0]], label [[DISPATCH:%.*]], label [[FINAL_RIGHT:%.*]]
-; CHECK: dispatch:
; CHECK-NEXT: [[V1_ADJ:%.*]] = add i8 [[V0]], [[V1:%.*]]
; CHECK-NEXT: [[C1:%.*]] = icmp eq i8 [[V1]], 0
-; CHECK-NEXT: br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT]]
+; CHECK-NEXT: [[OR_COND:%.*]] = and i1 [[C0]], [[C1]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT:%.*]]
; CHECK: final_left:
; CHECK-NEXT: call void @sideeffect0()
; CHECK-NEXT: call void @use8(i8 [[V1_ADJ]])
@@ -491,11 +495,10 @@ define void @one_pred_with_extra_op_externally_used_only_multiuse(i8 %v0, i8 %v1
; CHECK-LABEL: @one_pred_with_extra_op_externally_used_only_multiuse(
; CHECK-NEXT: pred:
; CHECK-NEXT: [[C0:%.*]] = icmp eq i8 [[V0:%.*]], 0
-; CHECK-NEXT: br i1 [[C0]], label [[DISPATCH:%.*]], label [[FINAL_RIGHT:%.*]]
-; CHECK: dispatch:
; CHECK-NEXT: [[V1_ADJ:%.*]] = add i8 [[V0]], [[V1:%.*]]
; CHECK-NEXT: [[C1:%.*]] = icmp eq i8 [[V1]], 0
-; CHECK-NEXT: br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT]]
+; CHECK-NEXT: [[OR_COND:%.*]] = and i1 [[C0]], [[C1]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[FINAL_LEFT:%.*]], label [[FINAL_RIGHT:%.*]]
; CHECK: final_left:
; CHECK-NEXT: call void @sideeffect0()
; CHECK-NEXT: call void @use8(i8 [[V1_ADJ]])
@@ -532,13 +535,16 @@ define void @two_preds_with_extra_op_externally_used_only(i8 %v0, i8 %v1, i8 %v2
; CHECK-NEXT: br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[DISPATCH:%.*]]
; CHECK: pred1:
; CHECK-NEXT: [[C2:%.*]] = icmp eq i8 [[V2:%.*]], 0
-; CHECK-NEXT: br i1 [[C2]], label [[DISPATCH]], label [[FINAL_RIGHT:%.*]]
-; CHECK: dispatch:
; CHECK-NEXT: [[V3_ADJ:%.*]] = add i8 [[V1]], [[V2]]
; CHECK-NEXT: [[C3:%.*]] = icmp eq i8 [[V3:%.*]], 0
-; CHECK-NEXT: br i1 [[C3]], label [[FINAL_LEFT]], label [[FINAL_RIGHT]]
+; CHECK-NEXT: [[OR_COND:%.*]] = and i1 [[C2]], [[C3]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[FINAL_LEFT]], label [[FINAL_RIGHT:%.*]]
+; CHECK: dispatch:
+; CHECK-NEXT: [[DOTOLD:%.*]] = add i8 [[V1]], [[V2]]
+; CHECK-NEXT: [[C3_OLD:%.*]] = icmp eq i8 [[V3]], 0
+; CHECK-NEXT: br i1 [[C3_OLD]], label [[FINAL_LEFT]], label [[FINAL_RIGHT]]
; CHECK: final_left:
-; CHECK-NEXT: [[MERGE_LEFT:%.*]] = phi i8 [ [[V3_ADJ]], [[DISPATCH]] ], [ 0, [[PRED0]] ]
+; CHECK-NEXT: [[MERGE_LEFT:%.*]] = phi i8 [ [[DOTOLD]], [[DISPATCH]] ], [ 0, [[PRED0]] ], [ [[V3_ADJ]], [[PRED1]] ]
; CHECK-NEXT: call void @use8(i8 [[MERGE_LEFT]])
; CHECK-NEXT: call void @sideeffect0()
; CHECK-NEXT: ret void
@@ -579,14 +585,17 @@ define void @two_preds_with_extra_op_externally_used_only_multiuse(i8 %v0, i8 %v
; CHECK-NEXT: br i1 [[C1]], label [[FINAL_LEFT:%.*]], label [[DISPATCH:%.*]]
; CHECK: pred1:
; CHECK-NEXT: [[C2:%.*]] = icmp eq i8 [[V2:%.*]], 0
-; CHECK-NEXT: br i1 [[C2]], label [[DISPATCH]], label [[FINAL_RIGHT:%.*]]
-; CHECK: dispatch:
; CHECK-NEXT: [[V3_ADJ:%.*]] = add i8 [[V1]], [[V2]]
; CHECK-NEXT: [[C3:%.*]] = icmp eq i8 [[V3:%.*]], 0
-; CHECK-NEXT: br i1 [[C3]], label [[FINAL_LEFT]], label [[FINAL_RIGHT]]
+; CHECK-NEXT: [[OR_COND:%.*]] = and i1 [[C2]], [[C3]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[FINAL_LEFT]], label [[FINAL_RIGHT:%.*]]
+; CHECK: dispatch:
+; CHECK-NEXT: [[DOTOLD:%.*]] = add i8 [[V1]], [[V2]]
+; CHECK-NEXT: [[C3_OLD:%.*]] = icmp eq i8 [[V3]], 0
+; CHECK-NEXT: br i1 [[C3_OLD]], label [[FINAL_LEFT]], label [[FINAL_RIGHT]]
; CHECK: final_left:
-; CHECK-NEXT: [[MERGE_LEFT:%.*]] = phi i8 [ [[V3_ADJ]], [[DISPATCH]] ], [ 0, [[PRED0]] ]
-; CHECK-NEXT: [[MERGE_LEFT_2:%.*]] = phi i8 [ [[V3_ADJ]], [[DISPATCH]] ], [ 42, [[PRED0]] ]
+; CHECK-NEXT: [[MERGE_LEFT:%.*]] = phi i8 [ [[DOTOLD]], [[DISPATCH]] ], [ 0, [[PRED0]] ], [ [[V3_ADJ]], [[PRED1]] ]
+; CHECK-NEXT: [[MERGE_LEFT_2:%.*]] = phi i8 [ [[DOTOLD]], [[DISPATCH]] ], [ 42, [[PRED0]] ], [ [[V3_ADJ]], [[PRED1]] ]
; CHECK-NEXT: call void @use8(i8 [[MERGE_LEFT]])
; CHECK-NEXT: call void @use8(i8 [[MERGE_LEFT_2]])
; CHECK-NEXT: call void @sideeffect0()
More information about the llvm-commits
mailing list