[llvm] [LoopUnroll] Penalize interior control flow (PR #67137)
Nikita Popov via llvm-commits
llvm-commits at lists.llvm.org
Fri Sep 22 06:41:56 PDT 2023
https://github.com/nikic created https://github.com/llvm/llvm-project/pull/67137
This patch adds a small cost penalty for each (non-trivial, non-exiting, non-latch) branch in the loop.
At a high level, I believe that fully unrolling loops with interior control flow tends to be less profitable, because we don't get straight-line (or extended-BB) code out of it, so it is less amenable to further optimization. It also dilutes branch predictors.
The specific motivation for this is twofold:
* This avoids an undesirable unroll in x264 that led to the revert of https://reviews.llvm.org/D156532. While the loop vectorizer can handle interior control flow, the SLP vectorizer can't (and likely won't be able to in any foreseeable future).
* While looking into https://github.com/llvm/llvm-project/issues/57865, I found a case where we produce a quadratic unroll result, because we first unroll an outer loop and then after simplification unroll (N copies of) the inner loop as well, resulting in huge IR output. The penalty applied here does not prevent this in general, but does disincentivize it. (We don't want to forbid unrolling outer loops entirely, it is necessary for vectorization in some cases.)
The penalty of 2 here is chosen somewhat arbitrarily as the lowest value that fixes both issues. I think the outer loop case may want an even larger penalty, but this seems like a conservative starting point.
>From cfbe72a6a8611a70bba1a5a41e7305a0b1565a65 Mon Sep 17 00:00:00 2001
From: Nikita Popov <npopov at redhat.com>
Date: Fri, 22 Sep 2023 15:13:43 +0200
Subject: [PATCH] [LoopUnroll] Penalize interior control flow
This patch adds a small cost penalty for each (non-trivial,
non-exiting, non-latch) branch in the loop.
At a high level, I believe that fully unrolling loops with interior
control flow tends to be less profitable, because we don't get
straight-line (or extended-BB) code out of it, so it is less
amenable to further optimization. It also dilutes branch predictors.
The specific motivation for this is twofold:
* This avoids an undesirable unroll in x264 that led to the revert
of https://reviews.llvm.org/D156532. While the loop vectorizer
can handle interior control flow, the SLP vectorizer can't (and
likely won't be able to in any foreseeable future).
* While looking into https://github.com/llvm/llvm-project/issues/57865,
I found a case where we produce a quadratic unroll result,
because we first unroll an outer loop and then after simplification
unroll (N copies of) the inner loop as well, resulting in huge
IR output. The penalty applied here does not prevent this
entirely, but does disincentivize it. (We don't want to forbid
unrolling outer loops entirely, it is necessary for vectorization
in some cases.)
The chosen penalty of 2 here is chosen somewhat arbitrarily as the
lowest value that fixes both issues. I think the outer loop case
may want an even larger penalty, but this seems like a conservative
starting point.
---
llvm/lib/Transforms/Scalar/LoopUnrollPass.cpp | 14 +
.../Transforms/LoopUnroll/quadratic-unroll.ll | 340 ++++++++++++++
.../runtime-loop-multiexit-dom-verify.ll | 2 +-
.../PhaseOrdering/AArch64/quant_4x4.ll | 444 +-----------------
4 files changed, 371 insertions(+), 429 deletions(-)
create mode 100644 llvm/test/Transforms/LoopUnroll/quadratic-unroll.ll
diff --git a/llvm/lib/Transforms/Scalar/LoopUnrollPass.cpp b/llvm/lib/Transforms/Scalar/LoopUnrollPass.cpp
index 335b489d3cb25ba..41c82783a7a4d79 100644
--- a/llvm/lib/Transforms/Scalar/LoopUnrollPass.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopUnrollPass.cpp
@@ -173,6 +173,10 @@ static cl::opt<unsigned>
cl::desc("Default threshold (max size of unrolled "
"loop), used in all but O3 optimizations"));
+static cl::opt<unsigned> UnrollInteriorControlFlowPenalty(
+ "unroll-interior-control-flow-penalty", cl::init(2), cl::Hidden,
+ cl::desc("Penalty for non-exiting branches in the loop"));
+
/// A magic value for use with the Threshold parameter to indicate
/// that the loop unroll should be performed regardless of how much
/// code expansion would result.
@@ -676,6 +680,16 @@ InstructionCost llvm::ApproximateLoopSize(
InstructionCost LoopSize = Metrics.NumInsts;
+ // Add a penalty for interior control flow (excluding exits and latches).
+ // Unrolling such loops is less profitable, as it does not result in
+ // straight-line code (or extended basic blocks if multiple exits). This
+ // also disincentivizes unrolling outer loops, which may result in large
+ // size increases if the inner loop is also unrolled later.
+ for (BasicBlock *BB : L->blocks())
+ if (!BB->getSingleSuccessor() && !L->isLoopExiting(BB) &&
+ !L->isLoopLatch(BB))
+ LoopSize += UnrollInteriorControlFlowPenalty;
+
// Don't allow an estimate of size zero. This would allows unrolling of loops
// with huge iteration counts, which is a compile time problem even if it's
// not a problem for code quality. Also, the code using this size may assume
diff --git a/llvm/test/Transforms/LoopUnroll/quadratic-unroll.ll b/llvm/test/Transforms/LoopUnroll/quadratic-unroll.ll
new file mode 100644
index 000000000000000..bfd88a12c9ca034
--- /dev/null
+++ b/llvm/test/Transforms/LoopUnroll/quadratic-unroll.ll
@@ -0,0 +1,340 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 3
+; RUN: opt -S -passes='loop(loop-unroll-full),simplifycfg,loop(loop-unroll-full)' -unroll-threshold=300 < %s | FileCheck %s
+
+; Check that we don't produce quadratic output by unrolling both the inner
+; and outer loop.
+
+ at a = global i32 0, align 4
+ at d = global ptr null, align 8
+ at e = external global ptr
+
+define i8 @f(ptr %p) {
+; CHECK-LABEL: define i8 @f(
+; CHECK-SAME: ptr [[P:%.*]]) {
+; CHECK-NEXT: entry:
+; CHECK-NEXT: br label [[FOR_COND1_PREHEADER:%.*]]
+; CHECK: for.cond1.preheader:
+; CHECK-NEXT: [[H_017:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INC12:%.*]], [[FOR_INC11:%.*]] ]
+; CHECK-NEXT: br label [[FOR_BODY4:%.*]]
+; CHECK: for.body4:
+; CHECK-NEXT: [[V:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT:%.*]] = icmp eq i32 [[V]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT]], label [[FOR_END:%.*]], label [[FOR_INC11]]
+; CHECK: for.end:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_1:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_1:%.*]] = icmp eq i32 [[V_1]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_1]], label [[FOR_END_1:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.1:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_2:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_2:%.*]] = icmp eq i32 [[V_2]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_2]], label [[FOR_END_2:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.2:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_3:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_3:%.*]] = icmp eq i32 [[V_3]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_3]], label [[FOR_END_3:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.3:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_4:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_4:%.*]] = icmp eq i32 [[V_4]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_4]], label [[FOR_END_4:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.4:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_5:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_5:%.*]] = icmp eq i32 [[V_5]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_5]], label [[FOR_END_5:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.5:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_6:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_6:%.*]] = icmp eq i32 [[V_6]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_6]], label [[FOR_END_6:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.6:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_7:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_7:%.*]] = icmp eq i32 [[V_7]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_7]], label [[FOR_END_7:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.7:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_8:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_8:%.*]] = icmp eq i32 [[V_8]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_8]], label [[FOR_END_8:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.8:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_9:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_9:%.*]] = icmp eq i32 [[V_9]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_9]], label [[FOR_END_9:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.9:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_10:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_10:%.*]] = icmp eq i32 [[V_10]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_10]], label [[FOR_END_10:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.10:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_11:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_11:%.*]] = icmp eq i32 [[V_11]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_11]], label [[FOR_END_11:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.11:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_12:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_12:%.*]] = icmp eq i32 [[V_12]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_12]], label [[FOR_END_12:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.12:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_13:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_13:%.*]] = icmp eq i32 [[V_13]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_13]], label [[FOR_END_13:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.13:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_14:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_14:%.*]] = icmp eq i32 [[V_14]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_14]], label [[FOR_END_14:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.14:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_15:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_15:%.*]] = icmp eq i32 [[V_15]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_15]], label [[FOR_END_15:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.15:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_16:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_16:%.*]] = icmp eq i32 [[V_16]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_16]], label [[FOR_END_16:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.16:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_17:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_17:%.*]] = icmp eq i32 [[V_17]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_17]], label [[FOR_END_17:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.17:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_18:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_18:%.*]] = icmp eq i32 [[V_18]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_18]], label [[FOR_END_18:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.18:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_19:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_19:%.*]] = icmp eq i32 [[V_19]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_19]], label [[FOR_END_19:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.19:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_20:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_20:%.*]] = icmp eq i32 [[V_20]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_20]], label [[FOR_END_20:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.20:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_21:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_21:%.*]] = icmp eq i32 [[V_21]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_21]], label [[FOR_END_21:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.21:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_22:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_22:%.*]] = icmp eq i32 [[V_22]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_22]], label [[FOR_END_22:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.22:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_23:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_23:%.*]] = icmp eq i32 [[V_23]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_23]], label [[FOR_END_23:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.23:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_24:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_24:%.*]] = icmp eq i32 [[V_24]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_24]], label [[FOR_END_24:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.24:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_25:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_25:%.*]] = icmp eq i32 [[V_25]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_25]], label [[FOR_END_25:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.25:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_26:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_26:%.*]] = icmp eq i32 [[V_26]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_26]], label [[FOR_END_26:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.26:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_27:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_27:%.*]] = icmp eq i32 [[V_27]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_27]], label [[FOR_END_27:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.27:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_28:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_28:%.*]] = icmp eq i32 [[V_28]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_28]], label [[FOR_END_28:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.28:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_29:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_29:%.*]] = icmp eq i32 [[V_29]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_29]], label [[FOR_END_29:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.29:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_30:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_30:%.*]] = icmp eq i32 [[V_30]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_30]], label [[FOR_END_30:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.30:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_31:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_31:%.*]] = icmp eq i32 [[V_31]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_31]], label [[FOR_END_31:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.31:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_32:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_32:%.*]] = icmp eq i32 [[V_32]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_32]], label [[FOR_END_32:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.32:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_33:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_33:%.*]] = icmp eq i32 [[V_33]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_33]], label [[FOR_END_33:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.33:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_34:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_34:%.*]] = icmp eq i32 [[V_34]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_34]], label [[FOR_END_34:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.34:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_35:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_35:%.*]] = icmp eq i32 [[V_35]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_35]], label [[FOR_END_35:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.35:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_36:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_36:%.*]] = icmp eq i32 [[V_36]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_36]], label [[FOR_END_36:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.36:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_37:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_37:%.*]] = icmp eq i32 [[V_37]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_37]], label [[FOR_END_37:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.37:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_38:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_38:%.*]] = icmp eq i32 [[V_38]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_38]], label [[FOR_END_38:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.38:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_39:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_39:%.*]] = icmp eq i32 [[V_39]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_39]], label [[FOR_END_39:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.39:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_40:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_40:%.*]] = icmp eq i32 [[V_40]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_40]], label [[FOR_END_40:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.40:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_41:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_41:%.*]] = icmp eq i32 [[V_41]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_41]], label [[FOR_END_41:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.41:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_42:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_42:%.*]] = icmp eq i32 [[V_42]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_42]], label [[FOR_END_42:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.42:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_43:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_43:%.*]] = icmp eq i32 [[V_43]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_43]], label [[FOR_END_43:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.43:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_44:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_44:%.*]] = icmp eq i32 [[V_44]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_44]], label [[FOR_END_44:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.44:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_45:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_45:%.*]] = icmp eq i32 [[V_45]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_45]], label [[FOR_END_45:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.45:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: [[V_46:%.*]] = load i32, ptr [[P]], align 4
+; CHECK-NEXT: [[TOBOOL_NOT_46:%.*]] = icmp eq i32 [[V_46]], 0
+; CHECK-NEXT: br i1 [[TOBOOL_NOT_46]], label [[FOR_END_46:%.*]], label [[FOR_INC11]]
+; CHECK: for.end.46:
+; CHECK-NEXT: store i32 0, ptr @a, align 4
+; CHECK-NEXT: store ptr @e, ptr @d, align 8
+; CHECK-NEXT: br label [[FOR_INC11]]
+; CHECK: for.inc11:
+; CHECK-NEXT: [[INC12]] = add nuw nsw i32 [[H_017]], 1
+; CHECK-NEXT: [[CMP:%.*]] = icmp ult i32 [[H_017]], 21
+; CHECK-NEXT: br i1 [[CMP]], label [[FOR_COND1_PREHEADER]], label [[FOR_END13:%.*]]
+; CHECK: for.end13:
+; CHECK-NEXT: ret i8 undef
+;
+entry:
+ br label %for.cond1.preheader
+
+for.cond1.preheader: ; preds = %entry, %for.inc11
+ %h.017 = phi i32 [ 0, %entry ], [ %inc12, %for.inc11 ]
+ br label %for.body4
+
+for.body4: ; preds = %for.cond1.preheader, %for.end
+ %g.016 = phi i16 [ -22, %for.cond1.preheader ], [ %inc9, %for.end ]
+ %v = load i32, ptr %p, align 4
+ %tobool.not = icmp eq i32 %v, 0
+ br i1 %tobool.not, label %for.cond5.preheader, label %for.inc11
+
+for.cond5.preheader: ; preds = %for.body4
+ br label %for.end
+
+for.end: ; preds = %for.cond5.preheader
+ store i32 0, ptr @a, align 4
+ store ptr @e, ptr @d, align 8
+ %inc9 = add i16 %g.016, 1
+ %cmp2 = icmp slt i16 %inc9, 25
+ br i1 %cmp2, label %for.body4, label %for.inc11
+
+for.inc11: ; preds = %for.end, %for.body4
+ %inc12 = add nuw nsw i32 %h.017, 1
+ %cmp = icmp ult i32 %h.017, 21
+ br i1 %cmp, label %for.cond1.preheader, label %for.end13
+
+for.end13: ; preds = %for.inc11
+ ret i8 undef
+}
diff --git a/llvm/test/Transforms/LoopUnroll/runtime-loop-multiexit-dom-verify.ll b/llvm/test/Transforms/LoopUnroll/runtime-loop-multiexit-dom-verify.ll
index de548523134568f..7027cac9c9ea355 100644
--- a/llvm/test/Transforms/LoopUnroll/runtime-loop-multiexit-dom-verify.ll
+++ b/llvm/test/Transforms/LoopUnroll/runtime-loop-multiexit-dom-verify.ll
@@ -1,5 +1,5 @@
; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=loop-unroll -unroll-runtime=true -unroll-runtime-epilog=false -unroll-runtime-multi-exit=true -unroll-count=4 -verify-dom-info -S | FileCheck %s
+; RUN: opt < %s -passes=loop-unroll -unroll-runtime=true -unroll-runtime-epilog=false -unroll-runtime-multi-exit=true -unroll-count=4 -unroll-threshold=300 -verify-dom-info -S | FileCheck %s
; REQUIRES: asserts
; The tests below are for verifying dom tree after runtime unrolling
diff --git a/llvm/test/Transforms/PhaseOrdering/AArch64/quant_4x4.ll b/llvm/test/Transforms/PhaseOrdering/AArch64/quant_4x4.ll
index d60b23279676748..abff36683d8717c 100644
--- a/llvm/test/Transforms/PhaseOrdering/AArch64/quant_4x4.ll
+++ b/llvm/test/Transforms/PhaseOrdering/AArch64/quant_4x4.ll
@@ -66,17 +66,22 @@ define i32 @quant_4x4(ptr noundef %dct, ptr noundef %mf, ptr noundef %bias) {
; CHECK-NEXT: [[TMP31:%.*]] = tail call i32 @llvm.vector.reduce.or.v8i32(<8 x i32> [[TMP30]])
; CHECK-NEXT: br label [[FOR_COND_CLEANUP:%.*]]
; CHECK: for.cond.cleanup:
-; CHECK-NEXT: [[OR_LCSSA:%.*]] = phi i32 [ [[TMP31]], [[VECTOR_BODY]] ], [ [[OR_15:%.*]], [[IF_END_15:%.*]] ]
+; CHECK-NEXT: [[OR_LCSSA:%.*]] = phi i32 [ [[TMP31]], [[VECTOR_BODY]] ], [ [[OR:%.*]], [[IF_END:%.*]] ]
; CHECK-NEXT: [[TOBOOL:%.*]] = icmp ne i32 [[OR_LCSSA]], 0
; CHECK-NEXT: [[LNOT_EXT:%.*]] = zext i1 [[TOBOOL]] to i32
; CHECK-NEXT: ret i32 [[LNOT_EXT]]
; CHECK: for.body:
-; CHECK-NEXT: [[TMP32:%.*]] = load i16, ptr [[DCT]], align 2
+; CHECK-NEXT: [[INDVARS_IV:%.*]] = phi i64 [ [[INDVARS_IV_NEXT:%.*]], [[IF_END]] ], [ 0, [[ENTRY:%.*]] ]
+; CHECK-NEXT: [[NZ_020:%.*]] = phi i32 [ [[OR]], [[IF_END]] ], [ 0, [[ENTRY]] ]
+; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds i16, ptr [[DCT]], i64 [[INDVARS_IV]]
+; CHECK-NEXT: [[TMP32:%.*]] = load i16, ptr [[ARRAYIDX]], align 2
; CHECK-NEXT: [[CONV:%.*]] = sext i16 [[TMP32]] to i32
; CHECK-NEXT: [[CMP1:%.*]] = icmp sgt i16 [[TMP32]], 0
-; CHECK-NEXT: [[TMP33:%.*]] = load i16, ptr [[BIAS]], align 2
+; CHECK-NEXT: [[ARRAYIDX4:%.*]] = getelementptr inbounds i16, ptr [[BIAS]], i64 [[INDVARS_IV]]
+; CHECK-NEXT: [[TMP33:%.*]] = load i16, ptr [[ARRAYIDX4]], align 2
; CHECK-NEXT: [[CONV5:%.*]] = zext i16 [[TMP33]] to i32
-; CHECK-NEXT: [[TMP34:%.*]] = load i16, ptr [[MF]], align 2
+; CHECK-NEXT: [[ARRAYIDX10:%.*]] = getelementptr inbounds i16, ptr [[MF]], i64 [[INDVARS_IV]]
+; CHECK-NEXT: [[TMP34:%.*]] = load i16, ptr [[ARRAYIDX10]], align 2
; CHECK-NEXT: [[CONV11:%.*]] = zext i16 [[TMP34]] to i32
; CHECK-NEXT: br i1 [[CMP1]], label [[IF_THEN:%.*]], label [[IF_ELSE:%.*]]
; CHECK: if.then:
@@ -84,7 +89,7 @@ define i32 @quant_4x4(ptr noundef %dct, ptr noundef %mf, ptr noundef %bias) {
; CHECK-NEXT: [[MUL:%.*]] = mul i32 [[ADD]], [[CONV11]]
; CHECK-NEXT: [[SHR:%.*]] = lshr i32 [[MUL]], 16
; CHECK-NEXT: [[CONV12:%.*]] = trunc i32 [[SHR]] to i16
-; CHECK-NEXT: br label [[IF_END:%.*]]
+; CHECK-NEXT: br label [[IF_END]]
; CHECK: if.else:
; CHECK-NEXT: [[ADD21:%.*]] = sub nsw i32 [[CONV5]], [[CONV]]
; CHECK-NEXT: [[MUL25:%.*]] = mul i32 [[ADD21]], [[CONV11]]
@@ -94,429 +99,12 @@ define i32 @quant_4x4(ptr noundef %dct, ptr noundef %mf, ptr noundef %bias) {
; CHECK-NEXT: br label [[IF_END]]
; CHECK: if.end:
; CHECK-NEXT: [[STOREMERGE:%.*]] = phi i16 [ [[CONV28]], [[IF_ELSE]] ], [ [[CONV12]], [[IF_THEN]] ]
-; CHECK-NEXT: store i16 [[STOREMERGE]], ptr [[DCT]], align 2
-; CHECK-NEXT: [[ARRAYIDX_1:%.*]] = getelementptr inbounds i16, ptr [[DCT]], i64 1
-; CHECK-NEXT: [[TMP36:%.*]] = load i16, ptr [[ARRAYIDX_1]], align 2
-; CHECK-NEXT: [[CONV_1:%.*]] = sext i16 [[TMP36]] to i32
-; CHECK-NEXT: [[CMP1_1:%.*]] = icmp sgt i16 [[TMP36]], 0
-; CHECK-NEXT: [[ARRAYIDX4_1:%.*]] = getelementptr inbounds i16, ptr [[BIAS]], i64 1
-; CHECK-NEXT: [[TMP37:%.*]] = load i16, ptr [[ARRAYIDX4_1]], align 2
-; CHECK-NEXT: [[CONV5_1:%.*]] = zext i16 [[TMP37]] to i32
-; CHECK-NEXT: [[ARRAYIDX10_1:%.*]] = getelementptr inbounds i16, ptr [[MF]], i64 1
-; CHECK-NEXT: [[TMP38:%.*]] = load i16, ptr [[ARRAYIDX10_1]], align 2
-; CHECK-NEXT: [[CONV11_1:%.*]] = zext i16 [[TMP38]] to i32
-; CHECK-NEXT: br i1 [[CMP1_1]], label [[IF_THEN_1:%.*]], label [[IF_ELSE_1:%.*]]
-; CHECK: if.else.1:
-; CHECK-NEXT: [[ADD21_1:%.*]] = sub nsw i32 [[CONV5_1]], [[CONV_1]]
-; CHECK-NEXT: [[MUL25_1:%.*]] = mul i32 [[ADD21_1]], [[CONV11_1]]
-; CHECK-NEXT: [[SHR26_1:%.*]] = lshr i32 [[MUL25_1]], 16
-; CHECK-NEXT: [[TMP39:%.*]] = trunc i32 [[SHR26_1]] to i16
-; CHECK-NEXT: [[CONV28_1:%.*]] = sub i16 0, [[TMP39]]
-; CHECK-NEXT: br label [[IF_END_1:%.*]]
-; CHECK: if.then.1:
-; CHECK-NEXT: [[ADD_1:%.*]] = add nuw nsw i32 [[CONV5_1]], [[CONV_1]]
-; CHECK-NEXT: [[MUL_1:%.*]] = mul i32 [[ADD_1]], [[CONV11_1]]
-; CHECK-NEXT: [[SHR_1:%.*]] = lshr i32 [[MUL_1]], 16
-; CHECK-NEXT: [[CONV12_1:%.*]] = trunc i32 [[SHR_1]] to i16
-; CHECK-NEXT: br label [[IF_END_1]]
-; CHECK: if.end.1:
-; CHECK-NEXT: [[STOREMERGE_1:%.*]] = phi i16 [ [[CONV28_1]], [[IF_ELSE_1]] ], [ [[CONV12_1]], [[IF_THEN_1]] ]
-; CHECK-NEXT: store i16 [[STOREMERGE_1]], ptr [[ARRAYIDX_1]], align 2
-; CHECK-NEXT: [[OR_131:%.*]] = or i16 [[STOREMERGE]], [[STOREMERGE_1]]
-; CHECK-NEXT: [[ARRAYIDX_2:%.*]] = getelementptr inbounds i16, ptr [[DCT]], i64 2
-; CHECK-NEXT: [[TMP40:%.*]] = load i16, ptr [[ARRAYIDX_2]], align 2
-; CHECK-NEXT: [[CONV_2:%.*]] = sext i16 [[TMP40]] to i32
-; CHECK-NEXT: [[CMP1_2:%.*]] = icmp sgt i16 [[TMP40]], 0
-; CHECK-NEXT: [[ARRAYIDX4_2:%.*]] = getelementptr inbounds i16, ptr [[BIAS]], i64 2
-; CHECK-NEXT: [[TMP41:%.*]] = load i16, ptr [[ARRAYIDX4_2]], align 2
-; CHECK-NEXT: [[CONV5_2:%.*]] = zext i16 [[TMP41]] to i32
-; CHECK-NEXT: [[ARRAYIDX10_2:%.*]] = getelementptr inbounds i16, ptr [[MF]], i64 2
-; CHECK-NEXT: [[TMP42:%.*]] = load i16, ptr [[ARRAYIDX10_2]], align 2
-; CHECK-NEXT: [[CONV11_2:%.*]] = zext i16 [[TMP42]] to i32
-; CHECK-NEXT: br i1 [[CMP1_2]], label [[IF_THEN_2:%.*]], label [[IF_ELSE_2:%.*]]
-; CHECK: if.else.2:
-; CHECK-NEXT: [[ADD21_2:%.*]] = sub nsw i32 [[CONV5_2]], [[CONV_2]]
-; CHECK-NEXT: [[MUL25_2:%.*]] = mul i32 [[ADD21_2]], [[CONV11_2]]
-; CHECK-NEXT: [[SHR26_2:%.*]] = lshr i32 [[MUL25_2]], 16
-; CHECK-NEXT: [[TMP43:%.*]] = trunc i32 [[SHR26_2]] to i16
-; CHECK-NEXT: [[CONV28_2:%.*]] = sub i16 0, [[TMP43]]
-; CHECK-NEXT: br label [[IF_END_2:%.*]]
-; CHECK: if.then.2:
-; CHECK-NEXT: [[ADD_2:%.*]] = add nuw nsw i32 [[CONV5_2]], [[CONV_2]]
-; CHECK-NEXT: [[MUL_2:%.*]] = mul i32 [[ADD_2]], [[CONV11_2]]
-; CHECK-NEXT: [[SHR_2:%.*]] = lshr i32 [[MUL_2]], 16
-; CHECK-NEXT: [[CONV12_2:%.*]] = trunc i32 [[SHR_2]] to i16
-; CHECK-NEXT: br label [[IF_END_2]]
-; CHECK: if.end.2:
-; CHECK-NEXT: [[STOREMERGE_2:%.*]] = phi i16 [ [[CONV28_2]], [[IF_ELSE_2]] ], [ [[CONV12_2]], [[IF_THEN_2]] ]
-; CHECK-NEXT: store i16 [[STOREMERGE_2]], ptr [[ARRAYIDX_2]], align 2
-; CHECK-NEXT: [[OR_232:%.*]] = or i16 [[OR_131]], [[STOREMERGE_2]]
-; CHECK-NEXT: [[ARRAYIDX_3:%.*]] = getelementptr inbounds i16, ptr [[DCT]], i64 3
-; CHECK-NEXT: [[TMP44:%.*]] = load i16, ptr [[ARRAYIDX_3]], align 2
-; CHECK-NEXT: [[CONV_3:%.*]] = sext i16 [[TMP44]] to i32
-; CHECK-NEXT: [[CMP1_3:%.*]] = icmp sgt i16 [[TMP44]], 0
-; CHECK-NEXT: [[ARRAYIDX4_3:%.*]] = getelementptr inbounds i16, ptr [[BIAS]], i64 3
-; CHECK-NEXT: [[TMP45:%.*]] = load i16, ptr [[ARRAYIDX4_3]], align 2
-; CHECK-NEXT: [[CONV5_3:%.*]] = zext i16 [[TMP45]] to i32
-; CHECK-NEXT: [[ARRAYIDX10_3:%.*]] = getelementptr inbounds i16, ptr [[MF]], i64 3
-; CHECK-NEXT: [[TMP46:%.*]] = load i16, ptr [[ARRAYIDX10_3]], align 2
-; CHECK-NEXT: [[CONV11_3:%.*]] = zext i16 [[TMP46]] to i32
-; CHECK-NEXT: br i1 [[CMP1_3]], label [[IF_THEN_3:%.*]], label [[IF_ELSE_3:%.*]]
-; CHECK: if.else.3:
-; CHECK-NEXT: [[ADD21_3:%.*]] = sub nsw i32 [[CONV5_3]], [[CONV_3]]
-; CHECK-NEXT: [[MUL25_3:%.*]] = mul i32 [[ADD21_3]], [[CONV11_3]]
-; CHECK-NEXT: [[SHR26_3:%.*]] = lshr i32 [[MUL25_3]], 16
-; CHECK-NEXT: [[TMP47:%.*]] = trunc i32 [[SHR26_3]] to i16
-; CHECK-NEXT: [[CONV28_3:%.*]] = sub i16 0, [[TMP47]]
-; CHECK-NEXT: br label [[IF_END_3:%.*]]
-; CHECK: if.then.3:
-; CHECK-NEXT: [[ADD_3:%.*]] = add nuw nsw i32 [[CONV5_3]], [[CONV_3]]
-; CHECK-NEXT: [[MUL_3:%.*]] = mul i32 [[ADD_3]], [[CONV11_3]]
-; CHECK-NEXT: [[SHR_3:%.*]] = lshr i32 [[MUL_3]], 16
-; CHECK-NEXT: [[CONV12_3:%.*]] = trunc i32 [[SHR_3]] to i16
-; CHECK-NEXT: br label [[IF_END_3]]
-; CHECK: if.end.3:
-; CHECK-NEXT: [[STOREMERGE_3:%.*]] = phi i16 [ [[CONV28_3]], [[IF_ELSE_3]] ], [ [[CONV12_3]], [[IF_THEN_3]] ]
-; CHECK-NEXT: store i16 [[STOREMERGE_3]], ptr [[ARRAYIDX_3]], align 2
-; CHECK-NEXT: [[OR_333:%.*]] = or i16 [[OR_232]], [[STOREMERGE_3]]
-; CHECK-NEXT: [[ARRAYIDX_4:%.*]] = getelementptr inbounds i16, ptr [[DCT]], i64 4
-; CHECK-NEXT: [[TMP48:%.*]] = load i16, ptr [[ARRAYIDX_4]], align 2
-; CHECK-NEXT: [[CONV_4:%.*]] = sext i16 [[TMP48]] to i32
-; CHECK-NEXT: [[CMP1_4:%.*]] = icmp sgt i16 [[TMP48]], 0
-; CHECK-NEXT: [[ARRAYIDX4_4:%.*]] = getelementptr inbounds i16, ptr [[BIAS]], i64 4
-; CHECK-NEXT: [[TMP49:%.*]] = load i16, ptr [[ARRAYIDX4_4]], align 2
-; CHECK-NEXT: [[CONV5_4:%.*]] = zext i16 [[TMP49]] to i32
-; CHECK-NEXT: [[ARRAYIDX10_4:%.*]] = getelementptr inbounds i16, ptr [[MF]], i64 4
-; CHECK-NEXT: [[TMP50:%.*]] = load i16, ptr [[ARRAYIDX10_4]], align 2
-; CHECK-NEXT: [[CONV11_4:%.*]] = zext i16 [[TMP50]] to i32
-; CHECK-NEXT: br i1 [[CMP1_4]], label [[IF_THEN_4:%.*]], label [[IF_ELSE_4:%.*]]
-; CHECK: if.else.4:
-; CHECK-NEXT: [[ADD21_4:%.*]] = sub nsw i32 [[CONV5_4]], [[CONV_4]]
-; CHECK-NEXT: [[MUL25_4:%.*]] = mul i32 [[ADD21_4]], [[CONV11_4]]
-; CHECK-NEXT: [[SHR26_4:%.*]] = lshr i32 [[MUL25_4]], 16
-; CHECK-NEXT: [[TMP51:%.*]] = trunc i32 [[SHR26_4]] to i16
-; CHECK-NEXT: [[CONV28_4:%.*]] = sub i16 0, [[TMP51]]
-; CHECK-NEXT: br label [[IF_END_4:%.*]]
-; CHECK: if.then.4:
-; CHECK-NEXT: [[ADD_4:%.*]] = add nuw nsw i32 [[CONV5_4]], [[CONV_4]]
-; CHECK-NEXT: [[MUL_4:%.*]] = mul i32 [[ADD_4]], [[CONV11_4]]
-; CHECK-NEXT: [[SHR_4:%.*]] = lshr i32 [[MUL_4]], 16
-; CHECK-NEXT: [[CONV12_4:%.*]] = trunc i32 [[SHR_4]] to i16
-; CHECK-NEXT: br label [[IF_END_4]]
-; CHECK: if.end.4:
-; CHECK-NEXT: [[STOREMERGE_4:%.*]] = phi i16 [ [[CONV28_4]], [[IF_ELSE_4]] ], [ [[CONV12_4]], [[IF_THEN_4]] ]
-; CHECK-NEXT: store i16 [[STOREMERGE_4]], ptr [[ARRAYIDX_4]], align 2
-; CHECK-NEXT: [[OR_434:%.*]] = or i16 [[OR_333]], [[STOREMERGE_4]]
-; CHECK-NEXT: [[ARRAYIDX_5:%.*]] = getelementptr inbounds i16, ptr [[DCT]], i64 5
-; CHECK-NEXT: [[TMP52:%.*]] = load i16, ptr [[ARRAYIDX_5]], align 2
-; CHECK-NEXT: [[CONV_5:%.*]] = sext i16 [[TMP52]] to i32
-; CHECK-NEXT: [[CMP1_5:%.*]] = icmp sgt i16 [[TMP52]], 0
-; CHECK-NEXT: [[ARRAYIDX4_5:%.*]] = getelementptr inbounds i16, ptr [[BIAS]], i64 5
-; CHECK-NEXT: [[TMP53:%.*]] = load i16, ptr [[ARRAYIDX4_5]], align 2
-; CHECK-NEXT: [[CONV5_5:%.*]] = zext i16 [[TMP53]] to i32
-; CHECK-NEXT: [[ARRAYIDX10_5:%.*]] = getelementptr inbounds i16, ptr [[MF]], i64 5
-; CHECK-NEXT: [[TMP54:%.*]] = load i16, ptr [[ARRAYIDX10_5]], align 2
-; CHECK-NEXT: [[CONV11_5:%.*]] = zext i16 [[TMP54]] to i32
-; CHECK-NEXT: br i1 [[CMP1_5]], label [[IF_THEN_5:%.*]], label [[IF_ELSE_5:%.*]]
-; CHECK: if.else.5:
-; CHECK-NEXT: [[ADD21_5:%.*]] = sub nsw i32 [[CONV5_5]], [[CONV_5]]
-; CHECK-NEXT: [[MUL25_5:%.*]] = mul i32 [[ADD21_5]], [[CONV11_5]]
-; CHECK-NEXT: [[SHR26_5:%.*]] = lshr i32 [[MUL25_5]], 16
-; CHECK-NEXT: [[TMP55:%.*]] = trunc i32 [[SHR26_5]] to i16
-; CHECK-NEXT: [[CONV28_5:%.*]] = sub i16 0, [[TMP55]]
-; CHECK-NEXT: br label [[IF_END_5:%.*]]
-; CHECK: if.then.5:
-; CHECK-NEXT: [[ADD_5:%.*]] = add nuw nsw i32 [[CONV5_5]], [[CONV_5]]
-; CHECK-NEXT: [[MUL_5:%.*]] = mul i32 [[ADD_5]], [[CONV11_5]]
-; CHECK-NEXT: [[SHR_5:%.*]] = lshr i32 [[MUL_5]], 16
-; CHECK-NEXT: [[CONV12_5:%.*]] = trunc i32 [[SHR_5]] to i16
-; CHECK-NEXT: br label [[IF_END_5]]
-; CHECK: if.end.5:
-; CHECK-NEXT: [[STOREMERGE_5:%.*]] = phi i16 [ [[CONV28_5]], [[IF_ELSE_5]] ], [ [[CONV12_5]], [[IF_THEN_5]] ]
-; CHECK-NEXT: store i16 [[STOREMERGE_5]], ptr [[ARRAYIDX_5]], align 2
-; CHECK-NEXT: [[OR_535:%.*]] = or i16 [[OR_434]], [[STOREMERGE_5]]
-; CHECK-NEXT: [[ARRAYIDX_6:%.*]] = getelementptr inbounds i16, ptr [[DCT]], i64 6
-; CHECK-NEXT: [[TMP56:%.*]] = load i16, ptr [[ARRAYIDX_6]], align 2
-; CHECK-NEXT: [[CONV_6:%.*]] = sext i16 [[TMP56]] to i32
-; CHECK-NEXT: [[CMP1_6:%.*]] = icmp sgt i16 [[TMP56]], 0
-; CHECK-NEXT: [[ARRAYIDX4_6:%.*]] = getelementptr inbounds i16, ptr [[BIAS]], i64 6
-; CHECK-NEXT: [[TMP57:%.*]] = load i16, ptr [[ARRAYIDX4_6]], align 2
-; CHECK-NEXT: [[CONV5_6:%.*]] = zext i16 [[TMP57]] to i32
-; CHECK-NEXT: [[ARRAYIDX10_6:%.*]] = getelementptr inbounds i16, ptr [[MF]], i64 6
-; CHECK-NEXT: [[TMP58:%.*]] = load i16, ptr [[ARRAYIDX10_6]], align 2
-; CHECK-NEXT: [[CONV11_6:%.*]] = zext i16 [[TMP58]] to i32
-; CHECK-NEXT: br i1 [[CMP1_6]], label [[IF_THEN_6:%.*]], label [[IF_ELSE_6:%.*]]
-; CHECK: if.else.6:
-; CHECK-NEXT: [[ADD21_6:%.*]] = sub nsw i32 [[CONV5_6]], [[CONV_6]]
-; CHECK-NEXT: [[MUL25_6:%.*]] = mul i32 [[ADD21_6]], [[CONV11_6]]
-; CHECK-NEXT: [[SHR26_6:%.*]] = lshr i32 [[MUL25_6]], 16
-; CHECK-NEXT: [[TMP59:%.*]] = trunc i32 [[SHR26_6]] to i16
-; CHECK-NEXT: [[CONV28_6:%.*]] = sub i16 0, [[TMP59]]
-; CHECK-NEXT: br label [[IF_END_6:%.*]]
-; CHECK: if.then.6:
-; CHECK-NEXT: [[ADD_6:%.*]] = add nuw nsw i32 [[CONV5_6]], [[CONV_6]]
-; CHECK-NEXT: [[MUL_6:%.*]] = mul i32 [[ADD_6]], [[CONV11_6]]
-; CHECK-NEXT: [[SHR_6:%.*]] = lshr i32 [[MUL_6]], 16
-; CHECK-NEXT: [[CONV12_6:%.*]] = trunc i32 [[SHR_6]] to i16
-; CHECK-NEXT: br label [[IF_END_6]]
-; CHECK: if.end.6:
-; CHECK-NEXT: [[STOREMERGE_6:%.*]] = phi i16 [ [[CONV28_6]], [[IF_ELSE_6]] ], [ [[CONV12_6]], [[IF_THEN_6]] ]
-; CHECK-NEXT: store i16 [[STOREMERGE_6]], ptr [[ARRAYIDX_6]], align 2
-; CHECK-NEXT: [[OR_636:%.*]] = or i16 [[OR_535]], [[STOREMERGE_6]]
-; CHECK-NEXT: [[ARRAYIDX_7:%.*]] = getelementptr inbounds i16, ptr [[DCT]], i64 7
-; CHECK-NEXT: [[TMP60:%.*]] = load i16, ptr [[ARRAYIDX_7]], align 2
-; CHECK-NEXT: [[CONV_7:%.*]] = sext i16 [[TMP60]] to i32
-; CHECK-NEXT: [[CMP1_7:%.*]] = icmp sgt i16 [[TMP60]], 0
-; CHECK-NEXT: [[ARRAYIDX4_7:%.*]] = getelementptr inbounds i16, ptr [[BIAS]], i64 7
-; CHECK-NEXT: [[TMP61:%.*]] = load i16, ptr [[ARRAYIDX4_7]], align 2
-; CHECK-NEXT: [[CONV5_7:%.*]] = zext i16 [[TMP61]] to i32
-; CHECK-NEXT: [[ARRAYIDX10_7:%.*]] = getelementptr inbounds i16, ptr [[MF]], i64 7
-; CHECK-NEXT: [[TMP62:%.*]] = load i16, ptr [[ARRAYIDX10_7]], align 2
-; CHECK-NEXT: [[CONV11_7:%.*]] = zext i16 [[TMP62]] to i32
-; CHECK-NEXT: br i1 [[CMP1_7]], label [[IF_THEN_7:%.*]], label [[IF_ELSE_7:%.*]]
-; CHECK: if.else.7:
-; CHECK-NEXT: [[ADD21_7:%.*]] = sub nsw i32 [[CONV5_7]], [[CONV_7]]
-; CHECK-NEXT: [[MUL25_7:%.*]] = mul i32 [[ADD21_7]], [[CONV11_7]]
-; CHECK-NEXT: [[SHR26_7:%.*]] = lshr i32 [[MUL25_7]], 16
-; CHECK-NEXT: [[TMP63:%.*]] = trunc i32 [[SHR26_7]] to i16
-; CHECK-NEXT: [[CONV28_7:%.*]] = sub i16 0, [[TMP63]]
-; CHECK-NEXT: br label [[IF_END_7:%.*]]
-; CHECK: if.then.7:
-; CHECK-NEXT: [[ADD_7:%.*]] = add nuw nsw i32 [[CONV5_7]], [[CONV_7]]
-; CHECK-NEXT: [[MUL_7:%.*]] = mul i32 [[ADD_7]], [[CONV11_7]]
-; CHECK-NEXT: [[SHR_7:%.*]] = lshr i32 [[MUL_7]], 16
-; CHECK-NEXT: [[CONV12_7:%.*]] = trunc i32 [[SHR_7]] to i16
-; CHECK-NEXT: br label [[IF_END_7]]
-; CHECK: if.end.7:
-; CHECK-NEXT: [[STOREMERGE_7:%.*]] = phi i16 [ [[CONV28_7]], [[IF_ELSE_7]] ], [ [[CONV12_7]], [[IF_THEN_7]] ]
-; CHECK-NEXT: store i16 [[STOREMERGE_7]], ptr [[ARRAYIDX_7]], align 2
-; CHECK-NEXT: [[OR_737:%.*]] = or i16 [[OR_636]], [[STOREMERGE_7]]
-; CHECK-NEXT: [[ARRAYIDX_8:%.*]] = getelementptr inbounds i16, ptr [[DCT]], i64 8
-; CHECK-NEXT: [[TMP64:%.*]] = load i16, ptr [[ARRAYIDX_8]], align 2
-; CHECK-NEXT: [[CONV_8:%.*]] = sext i16 [[TMP64]] to i32
-; CHECK-NEXT: [[CMP1_8:%.*]] = icmp sgt i16 [[TMP64]], 0
-; CHECK-NEXT: [[ARRAYIDX4_8:%.*]] = getelementptr inbounds i16, ptr [[BIAS]], i64 8
-; CHECK-NEXT: [[TMP65:%.*]] = load i16, ptr [[ARRAYIDX4_8]], align 2
-; CHECK-NEXT: [[CONV5_8:%.*]] = zext i16 [[TMP65]] to i32
-; CHECK-NEXT: [[ARRAYIDX10_8:%.*]] = getelementptr inbounds i16, ptr [[MF]], i64 8
-; CHECK-NEXT: [[TMP66:%.*]] = load i16, ptr [[ARRAYIDX10_8]], align 2
-; CHECK-NEXT: [[CONV11_8:%.*]] = zext i16 [[TMP66]] to i32
-; CHECK-NEXT: br i1 [[CMP1_8]], label [[IF_THEN_8:%.*]], label [[IF_ELSE_8:%.*]]
-; CHECK: if.else.8:
-; CHECK-NEXT: [[ADD21_8:%.*]] = sub nsw i32 [[CONV5_8]], [[CONV_8]]
-; CHECK-NEXT: [[MUL25_8:%.*]] = mul i32 [[ADD21_8]], [[CONV11_8]]
-; CHECK-NEXT: [[SHR26_8:%.*]] = lshr i32 [[MUL25_8]], 16
-; CHECK-NEXT: [[TMP67:%.*]] = trunc i32 [[SHR26_8]] to i16
-; CHECK-NEXT: [[CONV28_8:%.*]] = sub i16 0, [[TMP67]]
-; CHECK-NEXT: br label [[IF_END_8:%.*]]
-; CHECK: if.then.8:
-; CHECK-NEXT: [[ADD_8:%.*]] = add nuw nsw i32 [[CONV5_8]], [[CONV_8]]
-; CHECK-NEXT: [[MUL_8:%.*]] = mul i32 [[ADD_8]], [[CONV11_8]]
-; CHECK-NEXT: [[SHR_8:%.*]] = lshr i32 [[MUL_8]], 16
-; CHECK-NEXT: [[CONV12_8:%.*]] = trunc i32 [[SHR_8]] to i16
-; CHECK-NEXT: br label [[IF_END_8]]
-; CHECK: if.end.8:
-; CHECK-NEXT: [[STOREMERGE_8:%.*]] = phi i16 [ [[CONV28_8]], [[IF_ELSE_8]] ], [ [[CONV12_8]], [[IF_THEN_8]] ]
-; CHECK-NEXT: store i16 [[STOREMERGE_8]], ptr [[ARRAYIDX_8]], align 2
-; CHECK-NEXT: [[OR_838:%.*]] = or i16 [[OR_737]], [[STOREMERGE_8]]
-; CHECK-NEXT: [[ARRAYIDX_9:%.*]] = getelementptr inbounds i16, ptr [[DCT]], i64 9
-; CHECK-NEXT: [[TMP68:%.*]] = load i16, ptr [[ARRAYIDX_9]], align 2
-; CHECK-NEXT: [[CONV_9:%.*]] = sext i16 [[TMP68]] to i32
-; CHECK-NEXT: [[CMP1_9:%.*]] = icmp sgt i16 [[TMP68]], 0
-; CHECK-NEXT: [[ARRAYIDX4_9:%.*]] = getelementptr inbounds i16, ptr [[BIAS]], i64 9
-; CHECK-NEXT: [[TMP69:%.*]] = load i16, ptr [[ARRAYIDX4_9]], align 2
-; CHECK-NEXT: [[CONV5_9:%.*]] = zext i16 [[TMP69]] to i32
-; CHECK-NEXT: [[ARRAYIDX10_9:%.*]] = getelementptr inbounds i16, ptr [[MF]], i64 9
-; CHECK-NEXT: [[TMP70:%.*]] = load i16, ptr [[ARRAYIDX10_9]], align 2
-; CHECK-NEXT: [[CONV11_9:%.*]] = zext i16 [[TMP70]] to i32
-; CHECK-NEXT: br i1 [[CMP1_9]], label [[IF_THEN_9:%.*]], label [[IF_ELSE_9:%.*]]
-; CHECK: if.else.9:
-; CHECK-NEXT: [[ADD21_9:%.*]] = sub nsw i32 [[CONV5_9]], [[CONV_9]]
-; CHECK-NEXT: [[MUL25_9:%.*]] = mul i32 [[ADD21_9]], [[CONV11_9]]
-; CHECK-NEXT: [[SHR26_9:%.*]] = lshr i32 [[MUL25_9]], 16
-; CHECK-NEXT: [[TMP71:%.*]] = trunc i32 [[SHR26_9]] to i16
-; CHECK-NEXT: [[CONV28_9:%.*]] = sub i16 0, [[TMP71]]
-; CHECK-NEXT: br label [[IF_END_9:%.*]]
-; CHECK: if.then.9:
-; CHECK-NEXT: [[ADD_9:%.*]] = add nuw nsw i32 [[CONV5_9]], [[CONV_9]]
-; CHECK-NEXT: [[MUL_9:%.*]] = mul i32 [[ADD_9]], [[CONV11_9]]
-; CHECK-NEXT: [[SHR_9:%.*]] = lshr i32 [[MUL_9]], 16
-; CHECK-NEXT: [[CONV12_9:%.*]] = trunc i32 [[SHR_9]] to i16
-; CHECK-NEXT: br label [[IF_END_9]]
-; CHECK: if.end.9:
-; CHECK-NEXT: [[STOREMERGE_9:%.*]] = phi i16 [ [[CONV28_9]], [[IF_ELSE_9]] ], [ [[CONV12_9]], [[IF_THEN_9]] ]
-; CHECK-NEXT: store i16 [[STOREMERGE_9]], ptr [[ARRAYIDX_9]], align 2
-; CHECK-NEXT: [[OR_939:%.*]] = or i16 [[OR_838]], [[STOREMERGE_9]]
-; CHECK-NEXT: [[ARRAYIDX_10:%.*]] = getelementptr inbounds i16, ptr [[DCT]], i64 10
-; CHECK-NEXT: [[TMP72:%.*]] = load i16, ptr [[ARRAYIDX_10]], align 2
-; CHECK-NEXT: [[CONV_10:%.*]] = sext i16 [[TMP72]] to i32
-; CHECK-NEXT: [[CMP1_10:%.*]] = icmp sgt i16 [[TMP72]], 0
-; CHECK-NEXT: [[ARRAYIDX4_10:%.*]] = getelementptr inbounds i16, ptr [[BIAS]], i64 10
-; CHECK-NEXT: [[TMP73:%.*]] = load i16, ptr [[ARRAYIDX4_10]], align 2
-; CHECK-NEXT: [[CONV5_10:%.*]] = zext i16 [[TMP73]] to i32
-; CHECK-NEXT: [[ARRAYIDX10_10:%.*]] = getelementptr inbounds i16, ptr [[MF]], i64 10
-; CHECK-NEXT: [[TMP74:%.*]] = load i16, ptr [[ARRAYIDX10_10]], align 2
-; CHECK-NEXT: [[CONV11_10:%.*]] = zext i16 [[TMP74]] to i32
-; CHECK-NEXT: br i1 [[CMP1_10]], label [[IF_THEN_10:%.*]], label [[IF_ELSE_10:%.*]]
-; CHECK: if.else.10:
-; CHECK-NEXT: [[ADD21_10:%.*]] = sub nsw i32 [[CONV5_10]], [[CONV_10]]
-; CHECK-NEXT: [[MUL25_10:%.*]] = mul i32 [[ADD21_10]], [[CONV11_10]]
-; CHECK-NEXT: [[SHR26_10:%.*]] = lshr i32 [[MUL25_10]], 16
-; CHECK-NEXT: [[TMP75:%.*]] = trunc i32 [[SHR26_10]] to i16
-; CHECK-NEXT: [[CONV28_10:%.*]] = sub i16 0, [[TMP75]]
-; CHECK-NEXT: br label [[IF_END_10:%.*]]
-; CHECK: if.then.10:
-; CHECK-NEXT: [[ADD_10:%.*]] = add nuw nsw i32 [[CONV5_10]], [[CONV_10]]
-; CHECK-NEXT: [[MUL_10:%.*]] = mul i32 [[ADD_10]], [[CONV11_10]]
-; CHECK-NEXT: [[SHR_10:%.*]] = lshr i32 [[MUL_10]], 16
-; CHECK-NEXT: [[CONV12_10:%.*]] = trunc i32 [[SHR_10]] to i16
-; CHECK-NEXT: br label [[IF_END_10]]
-; CHECK: if.end.10:
-; CHECK-NEXT: [[STOREMERGE_10:%.*]] = phi i16 [ [[CONV28_10]], [[IF_ELSE_10]] ], [ [[CONV12_10]], [[IF_THEN_10]] ]
-; CHECK-NEXT: store i16 [[STOREMERGE_10]], ptr [[ARRAYIDX_10]], align 2
-; CHECK-NEXT: [[OR_1040:%.*]] = or i16 [[OR_939]], [[STOREMERGE_10]]
-; CHECK-NEXT: [[ARRAYIDX_11:%.*]] = getelementptr inbounds i16, ptr [[DCT]], i64 11
-; CHECK-NEXT: [[TMP76:%.*]] = load i16, ptr [[ARRAYIDX_11]], align 2
-; CHECK-NEXT: [[CONV_11:%.*]] = sext i16 [[TMP76]] to i32
-; CHECK-NEXT: [[CMP1_11:%.*]] = icmp sgt i16 [[TMP76]], 0
-; CHECK-NEXT: [[ARRAYIDX4_11:%.*]] = getelementptr inbounds i16, ptr [[BIAS]], i64 11
-; CHECK-NEXT: [[TMP77:%.*]] = load i16, ptr [[ARRAYIDX4_11]], align 2
-; CHECK-NEXT: [[CONV5_11:%.*]] = zext i16 [[TMP77]] to i32
-; CHECK-NEXT: [[ARRAYIDX10_11:%.*]] = getelementptr inbounds i16, ptr [[MF]], i64 11
-; CHECK-NEXT: [[TMP78:%.*]] = load i16, ptr [[ARRAYIDX10_11]], align 2
-; CHECK-NEXT: [[CONV11_11:%.*]] = zext i16 [[TMP78]] to i32
-; CHECK-NEXT: br i1 [[CMP1_11]], label [[IF_THEN_11:%.*]], label [[IF_ELSE_11:%.*]]
-; CHECK: if.else.11:
-; CHECK-NEXT: [[ADD21_11:%.*]] = sub nsw i32 [[CONV5_11]], [[CONV_11]]
-; CHECK-NEXT: [[MUL25_11:%.*]] = mul i32 [[ADD21_11]], [[CONV11_11]]
-; CHECK-NEXT: [[SHR26_11:%.*]] = lshr i32 [[MUL25_11]], 16
-; CHECK-NEXT: [[TMP79:%.*]] = trunc i32 [[SHR26_11]] to i16
-; CHECK-NEXT: [[CONV28_11:%.*]] = sub i16 0, [[TMP79]]
-; CHECK-NEXT: br label [[IF_END_11:%.*]]
-; CHECK: if.then.11:
-; CHECK-NEXT: [[ADD_11:%.*]] = add nuw nsw i32 [[CONV5_11]], [[CONV_11]]
-; CHECK-NEXT: [[MUL_11:%.*]] = mul i32 [[ADD_11]], [[CONV11_11]]
-; CHECK-NEXT: [[SHR_11:%.*]] = lshr i32 [[MUL_11]], 16
-; CHECK-NEXT: [[CONV12_11:%.*]] = trunc i32 [[SHR_11]] to i16
-; CHECK-NEXT: br label [[IF_END_11]]
-; CHECK: if.end.11:
-; CHECK-NEXT: [[STOREMERGE_11:%.*]] = phi i16 [ [[CONV28_11]], [[IF_ELSE_11]] ], [ [[CONV12_11]], [[IF_THEN_11]] ]
-; CHECK-NEXT: store i16 [[STOREMERGE_11]], ptr [[ARRAYIDX_11]], align 2
-; CHECK-NEXT: [[OR_1141:%.*]] = or i16 [[OR_1040]], [[STOREMERGE_11]]
-; CHECK-NEXT: [[ARRAYIDX_12:%.*]] = getelementptr inbounds i16, ptr [[DCT]], i64 12
-; CHECK-NEXT: [[TMP80:%.*]] = load i16, ptr [[ARRAYIDX_12]], align 2
-; CHECK-NEXT: [[CONV_12:%.*]] = sext i16 [[TMP80]] to i32
-; CHECK-NEXT: [[CMP1_12:%.*]] = icmp sgt i16 [[TMP80]], 0
-; CHECK-NEXT: [[ARRAYIDX4_12:%.*]] = getelementptr inbounds i16, ptr [[BIAS]], i64 12
-; CHECK-NEXT: [[TMP81:%.*]] = load i16, ptr [[ARRAYIDX4_12]], align 2
-; CHECK-NEXT: [[CONV5_12:%.*]] = zext i16 [[TMP81]] to i32
-; CHECK-NEXT: [[ARRAYIDX10_12:%.*]] = getelementptr inbounds i16, ptr [[MF]], i64 12
-; CHECK-NEXT: [[TMP82:%.*]] = load i16, ptr [[ARRAYIDX10_12]], align 2
-; CHECK-NEXT: [[CONV11_12:%.*]] = zext i16 [[TMP82]] to i32
-; CHECK-NEXT: br i1 [[CMP1_12]], label [[IF_THEN_12:%.*]], label [[IF_ELSE_12:%.*]]
-; CHECK: if.else.12:
-; CHECK-NEXT: [[ADD21_12:%.*]] = sub nsw i32 [[CONV5_12]], [[CONV_12]]
-; CHECK-NEXT: [[MUL25_12:%.*]] = mul i32 [[ADD21_12]], [[CONV11_12]]
-; CHECK-NEXT: [[SHR26_12:%.*]] = lshr i32 [[MUL25_12]], 16
-; CHECK-NEXT: [[TMP83:%.*]] = trunc i32 [[SHR26_12]] to i16
-; CHECK-NEXT: [[CONV28_12:%.*]] = sub i16 0, [[TMP83]]
-; CHECK-NEXT: br label [[IF_END_12:%.*]]
-; CHECK: if.then.12:
-; CHECK-NEXT: [[ADD_12:%.*]] = add nuw nsw i32 [[CONV5_12]], [[CONV_12]]
-; CHECK-NEXT: [[MUL_12:%.*]] = mul i32 [[ADD_12]], [[CONV11_12]]
-; CHECK-NEXT: [[SHR_12:%.*]] = lshr i32 [[MUL_12]], 16
-; CHECK-NEXT: [[CONV12_12:%.*]] = trunc i32 [[SHR_12]] to i16
-; CHECK-NEXT: br label [[IF_END_12]]
-; CHECK: if.end.12:
-; CHECK-NEXT: [[STOREMERGE_12:%.*]] = phi i16 [ [[CONV28_12]], [[IF_ELSE_12]] ], [ [[CONV12_12]], [[IF_THEN_12]] ]
-; CHECK-NEXT: store i16 [[STOREMERGE_12]], ptr [[ARRAYIDX_12]], align 2
-; CHECK-NEXT: [[OR_1242:%.*]] = or i16 [[OR_1141]], [[STOREMERGE_12]]
-; CHECK-NEXT: [[ARRAYIDX_13:%.*]] = getelementptr inbounds i16, ptr [[DCT]], i64 13
-; CHECK-NEXT: [[TMP84:%.*]] = load i16, ptr [[ARRAYIDX_13]], align 2
-; CHECK-NEXT: [[CONV_13:%.*]] = sext i16 [[TMP84]] to i32
-; CHECK-NEXT: [[CMP1_13:%.*]] = icmp sgt i16 [[TMP84]], 0
-; CHECK-NEXT: [[ARRAYIDX4_13:%.*]] = getelementptr inbounds i16, ptr [[BIAS]], i64 13
-; CHECK-NEXT: [[TMP85:%.*]] = load i16, ptr [[ARRAYIDX4_13]], align 2
-; CHECK-NEXT: [[CONV5_13:%.*]] = zext i16 [[TMP85]] to i32
-; CHECK-NEXT: [[ARRAYIDX10_13:%.*]] = getelementptr inbounds i16, ptr [[MF]], i64 13
-; CHECK-NEXT: [[TMP86:%.*]] = load i16, ptr [[ARRAYIDX10_13]], align 2
-; CHECK-NEXT: [[CONV11_13:%.*]] = zext i16 [[TMP86]] to i32
-; CHECK-NEXT: br i1 [[CMP1_13]], label [[IF_THEN_13:%.*]], label [[IF_ELSE_13:%.*]]
-; CHECK: if.else.13:
-; CHECK-NEXT: [[ADD21_13:%.*]] = sub nsw i32 [[CONV5_13]], [[CONV_13]]
-; CHECK-NEXT: [[MUL25_13:%.*]] = mul i32 [[ADD21_13]], [[CONV11_13]]
-; CHECK-NEXT: [[SHR26_13:%.*]] = lshr i32 [[MUL25_13]], 16
-; CHECK-NEXT: [[TMP87:%.*]] = trunc i32 [[SHR26_13]] to i16
-; CHECK-NEXT: [[CONV28_13:%.*]] = sub i16 0, [[TMP87]]
-; CHECK-NEXT: br label [[IF_END_13:%.*]]
-; CHECK: if.then.13:
-; CHECK-NEXT: [[ADD_13:%.*]] = add nuw nsw i32 [[CONV5_13]], [[CONV_13]]
-; CHECK-NEXT: [[MUL_13:%.*]] = mul i32 [[ADD_13]], [[CONV11_13]]
-; CHECK-NEXT: [[SHR_13:%.*]] = lshr i32 [[MUL_13]], 16
-; CHECK-NEXT: [[CONV12_13:%.*]] = trunc i32 [[SHR_13]] to i16
-; CHECK-NEXT: br label [[IF_END_13]]
-; CHECK: if.end.13:
-; CHECK-NEXT: [[STOREMERGE_13:%.*]] = phi i16 [ [[CONV28_13]], [[IF_ELSE_13]] ], [ [[CONV12_13]], [[IF_THEN_13]] ]
-; CHECK-NEXT: store i16 [[STOREMERGE_13]], ptr [[ARRAYIDX_13]], align 2
-; CHECK-NEXT: [[OR_1343:%.*]] = or i16 [[OR_1242]], [[STOREMERGE_13]]
-; CHECK-NEXT: [[ARRAYIDX_14:%.*]] = getelementptr inbounds i16, ptr [[DCT]], i64 14
-; CHECK-NEXT: [[TMP88:%.*]] = load i16, ptr [[ARRAYIDX_14]], align 2
-; CHECK-NEXT: [[CONV_14:%.*]] = sext i16 [[TMP88]] to i32
-; CHECK-NEXT: [[CMP1_14:%.*]] = icmp sgt i16 [[TMP88]], 0
-; CHECK-NEXT: [[ARRAYIDX4_14:%.*]] = getelementptr inbounds i16, ptr [[BIAS]], i64 14
-; CHECK-NEXT: [[TMP89:%.*]] = load i16, ptr [[ARRAYIDX4_14]], align 2
-; CHECK-NEXT: [[CONV5_14:%.*]] = zext i16 [[TMP89]] to i32
-; CHECK-NEXT: [[ARRAYIDX10_14:%.*]] = getelementptr inbounds i16, ptr [[MF]], i64 14
-; CHECK-NEXT: [[TMP90:%.*]] = load i16, ptr [[ARRAYIDX10_14]], align 2
-; CHECK-NEXT: [[CONV11_14:%.*]] = zext i16 [[TMP90]] to i32
-; CHECK-NEXT: br i1 [[CMP1_14]], label [[IF_THEN_14:%.*]], label [[IF_ELSE_14:%.*]]
-; CHECK: if.else.14:
-; CHECK-NEXT: [[ADD21_14:%.*]] = sub nsw i32 [[CONV5_14]], [[CONV_14]]
-; CHECK-NEXT: [[MUL25_14:%.*]] = mul i32 [[ADD21_14]], [[CONV11_14]]
-; CHECK-NEXT: [[SHR26_14:%.*]] = lshr i32 [[MUL25_14]], 16
-; CHECK-NEXT: [[TMP91:%.*]] = trunc i32 [[SHR26_14]] to i16
-; CHECK-NEXT: [[CONV28_14:%.*]] = sub i16 0, [[TMP91]]
-; CHECK-NEXT: br label [[IF_END_14:%.*]]
-; CHECK: if.then.14:
-; CHECK-NEXT: [[ADD_14:%.*]] = add nuw nsw i32 [[CONV5_14]], [[CONV_14]]
-; CHECK-NEXT: [[MUL_14:%.*]] = mul i32 [[ADD_14]], [[CONV11_14]]
-; CHECK-NEXT: [[SHR_14:%.*]] = lshr i32 [[MUL_14]], 16
-; CHECK-NEXT: [[CONV12_14:%.*]] = trunc i32 [[SHR_14]] to i16
-; CHECK-NEXT: br label [[IF_END_14]]
-; CHECK: if.end.14:
-; CHECK-NEXT: [[STOREMERGE_14:%.*]] = phi i16 [ [[CONV28_14]], [[IF_ELSE_14]] ], [ [[CONV12_14]], [[IF_THEN_14]] ]
-; CHECK-NEXT: store i16 [[STOREMERGE_14]], ptr [[ARRAYIDX_14]], align 2
-; CHECK-NEXT: [[OR_1444:%.*]] = or i16 [[OR_1343]], [[STOREMERGE_14]]
-; CHECK-NEXT: [[ARRAYIDX_15:%.*]] = getelementptr inbounds i16, ptr [[DCT]], i64 15
-; CHECK-NEXT: [[TMP92:%.*]] = load i16, ptr [[ARRAYIDX_15]], align 2
-; CHECK-NEXT: [[CONV_15:%.*]] = sext i16 [[TMP92]] to i32
-; CHECK-NEXT: [[CMP1_15:%.*]] = icmp sgt i16 [[TMP92]], 0
-; CHECK-NEXT: [[ARRAYIDX4_15:%.*]] = getelementptr inbounds i16, ptr [[BIAS]], i64 15
-; CHECK-NEXT: [[TMP93:%.*]] = load i16, ptr [[ARRAYIDX4_15]], align 2
-; CHECK-NEXT: [[CONV5_15:%.*]] = zext i16 [[TMP93]] to i32
-; CHECK-NEXT: [[ARRAYIDX10_15:%.*]] = getelementptr inbounds i16, ptr [[MF]], i64 15
-; CHECK-NEXT: [[TMP94:%.*]] = load i16, ptr [[ARRAYIDX10_15]], align 2
-; CHECK-NEXT: [[CONV11_15:%.*]] = zext i16 [[TMP94]] to i32
-; CHECK-NEXT: br i1 [[CMP1_15]], label [[IF_THEN_15:%.*]], label [[IF_ELSE_15:%.*]]
-; CHECK: if.else.15:
-; CHECK-NEXT: [[ADD21_15:%.*]] = sub nsw i32 [[CONV5_15]], [[CONV_15]]
-; CHECK-NEXT: [[MUL25_15:%.*]] = mul i32 [[ADD21_15]], [[CONV11_15]]
-; CHECK-NEXT: [[SHR26_15:%.*]] = lshr i32 [[MUL25_15]], 16
-; CHECK-NEXT: [[TMP95:%.*]] = trunc i32 [[SHR26_15]] to i16
-; CHECK-NEXT: [[CONV28_15:%.*]] = sub i16 0, [[TMP95]]
-; CHECK-NEXT: br label [[IF_END_15]]
-; CHECK: if.then.15:
-; CHECK-NEXT: [[ADD_15:%.*]] = add nuw nsw i32 [[CONV5_15]], [[CONV_15]]
-; CHECK-NEXT: [[MUL_15:%.*]] = mul i32 [[ADD_15]], [[CONV11_15]]
-; CHECK-NEXT: [[SHR_15:%.*]] = lshr i32 [[MUL_15]], 16
-; CHECK-NEXT: [[CONV12_15:%.*]] = trunc i32 [[SHR_15]] to i16
-; CHECK-NEXT: br label [[IF_END_15]]
-; CHECK: if.end.15:
-; CHECK-NEXT: [[STOREMERGE_15:%.*]] = phi i16 [ [[CONV28_15]], [[IF_ELSE_15]] ], [ [[CONV12_15]], [[IF_THEN_15]] ]
-; CHECK-NEXT: store i16 [[STOREMERGE_15]], ptr [[ARRAYIDX_15]], align 2
-; CHECK-NEXT: [[OR_1545:%.*]] = or i16 [[OR_1444]], [[STOREMERGE_15]]
-; CHECK-NEXT: [[OR_15]] = sext i16 [[OR_1545]] to i32
-; CHECK-NEXT: br label [[FOR_COND_CLEANUP]]
+; CHECK-NEXT: store i16 [[STOREMERGE]], ptr [[ARRAYIDX]], align 2
+; CHECK-NEXT: [[CONV33:%.*]] = sext i16 [[STOREMERGE]] to i32
+; CHECK-NEXT: [[OR]] = or i32 [[NZ_020]], [[CONV33]]
+; CHECK-NEXT: [[INDVARS_IV_NEXT]] = add nuw nsw i64 [[INDVARS_IV]], 1
+; CHECK-NEXT: [[EXITCOND_NOT:%.*]] = icmp eq i64 [[INDVARS_IV_NEXT]], 16
+; CHECK-NEXT: br i1 [[EXITCOND_NOT]], label [[FOR_COND_CLEANUP]], label [[FOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
;
entry:
%dct.addr = alloca ptr, align 8
More information about the llvm-commits
mailing list