[llvm] 62ea799 - [AMDGPU] Break Large PHIs: Take whole PHI chains into account
via llvm-commits
llvm-commits at lists.llvm.org
Thu Aug 3 07:41:16 PDT 2023
Author: pvanhout
Date: 2023-08-03T16:41:11+02:00
New Revision: 62ea799e6c798e84428d77bc84b83fc572b7d75e
URL: https://github.com/llvm/llvm-project/commit/62ea799e6c798e84428d77bc84b83fc572b7d75e
DIFF: https://github.com/llvm/llvm-project/commit/62ea799e6c798e84428d77bc84b83fc572b7d75e.diff
LOG: [AMDGPU] Break Large PHIs: Take whole PHI chains into account
Previous heuristics had a big flaw: they only looked at single PHI at a time, and didn't take into account the whole "chain".
The concept of "chain" is important because if we only break a chain partially, we risk forcing regalloc to reserve twice as many registers for that vector.
We also risk adding a lot of copies that shouldn't be there and can inhibit backend optimizations.
The solution I found is to consider the whole "PHI chain" when looking at PHI.
That is, we recursively look at the PHI's incoming value & users for other PHIs, then make a decision about the chain as a whole.
The currrent threshold requires that at least `ceil(chain size * (2/3))` PHIs have at least one interesting incoming value.
In simple terms, two-thirds (rounded up) of the PHIs should be breakable.
This seems to work well. A lower threshold such as 50% is too aggressive because chains can often have 7 or 9 PHIs, and breaking 3+ or 4+ PHIs in those case often causes performance issue.
Fixes SWDEV-409648, SWDEV-398393, SWDEV-413487
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D156414
Added:
Modified:
llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp
llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-break-large-phis-heuristics.ll
Removed:
################################################################################
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp b/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp
index 4ec85f3c55884c..5cb4754fb76e3c 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp
@@ -52,17 +52,17 @@ static cl::opt<bool> Widen16BitOps(
cl::init(true));
static cl::opt<bool>
- ScalarizeLargePHIs("amdgpu-codegenprepare-break-large-phis",
- cl::desc("Break large PHI nodes for DAGISel"),
- cl::ReallyHidden, cl::init(true));
+ BreakLargePHIs("amdgpu-codegenprepare-break-large-phis",
+ cl::desc("Break large PHI nodes for DAGISel"),
+ cl::ReallyHidden, cl::init(true));
static cl::opt<bool>
- ForceScalarizeLargePHIs("amdgpu-codegenprepare-force-break-large-phis",
- cl::desc("For testing purposes, always break large "
- "PHIs even if it isn't profitable."),
- cl::ReallyHidden, cl::init(false));
+ ForceBreakLargePHIs("amdgpu-codegenprepare-force-break-large-phis",
+ cl::desc("For testing purposes, always break large "
+ "PHIs even if it isn't profitable."),
+ cl::ReallyHidden, cl::init(false));
-static cl::opt<unsigned> ScalarizeLargePHIsThreshold(
+static cl::opt<unsigned> BreakLargePHIsThreshold(
"amdgpu-codegenprepare-break-large-phis-threshold",
cl::desc("Minimum type size in bits for breaking large PHI nodes"),
cl::ReallyHidden, cl::init(32));
@@ -1777,47 +1777,79 @@ static bool isInterestingPHIIncomingValue(const Value *V) {
return false;
}
+static void collectPHINodes(const PHINode &I,
+ SmallPtrSet<const PHINode *, 8> &SeenPHIs) {
+ const auto [It, Inserted] = SeenPHIs.insert(&I);
+ if (!Inserted)
+ return;
+
+ for (const Value *Inc : I.incoming_values()) {
+ if (const auto *PhiInc = dyn_cast<PHINode>(Inc))
+ collectPHINodes(*PhiInc, SeenPHIs);
+ }
+
+ for (const User *U : I.users()) {
+ if (const auto *PhiU = dyn_cast<PHINode>(U))
+ collectPHINodes(*PhiU, SeenPHIs);
+ }
+}
+
bool AMDGPUCodeGenPrepareImpl::canBreakPHINode(const PHINode &I) {
- // Check in the cache, or add an entry for this node.
- //
- // We init with false because we consider all PHI nodes unbreakable until we
- // reach a conclusion. Doing the opposite - assuming they're break-able until
- // proven otherwise - can be harmful in some pathological cases so we're
- // conservative for now.
- const auto [It, DidInsert] = BreakPhiNodesCache.insert({&I, false});
- if (!DidInsert)
+ // Check in the cache first.
+ if (const auto It = BreakPhiNodesCache.find(&I);
+ It != BreakPhiNodesCache.end())
return It->second;
- // This function may recurse, so to guard against infinite looping, this PHI
- // is conservatively considered unbreakable until we reach a conclusion.
+ // We consider PHI nodes as part of "chains", so given a PHI node I, we
+ // recursively consider all its users and incoming values that are also PHI
+ // nodes. We then make a decision about all of those PHIs at once. Either they
+ // all get broken up, or none of them do. That way, we avoid cases where a
+ // single PHI is/is not broken and we end up reforming/exploding a vector
+ // multiple times, or even worse, doing it in a loop.
+ SmallPtrSet<const PHINode *, 8> WorkList;
+ collectPHINodes(I, WorkList);
+
+#ifndef NDEBUG
+ // Check that none of the PHI nodes in the worklist are in the map. If some of
+ // them are, it means we're not good enough at collecting related PHIs.
+ for (const PHINode *WLP : WorkList) {
+ assert(BreakPhiNodesCache.count(WLP) == 0);
+ }
+#endif
- // Don't break PHIs that have no interesting incoming values. That is, where
- // there is no clear opportunity to fold the "extractelement" instructions we
- // would add.
+ // To consider a PHI profitable to break, we need to see some interesting
+ // incoming values. At least 2/3rd (rounded up) of all PHIs in the worklist
+ // must have one to consider all PHIs breakable.
//
- // Note: IC does not run after this pass, so we're only interested in the
- // foldings that the DAG combiner can do.
- if (none_of(I.incoming_values(),
- [&](Value *V) { return isInterestingPHIIncomingValue(V); }))
- return false;
-
- // Now, check users for unbreakable PHI nodes. If we have an unbreakable PHI
- // node as user, we don't want to break this PHI either because it's unlikely
- // to be beneficial. We would just explode the vector and reassemble it
- // directly, wasting instructions.
+ // This threshold has been determined through performance testing.
+ //
+ // Note that the computation below is equivalent to
+ //
+ // (unsigned)ceil((K / 3.0) * 2)
//
- // In the case where multiple users are PHI nodes, we want at least half of
- // them to be breakable.
- int Score = 0;
- for (const Value *U : I.users()) {
- if (const auto *PU = dyn_cast<PHINode>(U))
- Score += canBreakPHINode(*PU) ? 1 : -1;
+ // It's simply written this way to avoid mixing integral/FP arithmetic.
+ const auto Threshold = (alignTo(WorkList.size() * 2, 3) / 3);
+ unsigned NumBreakablePHIs = 0;
+ bool CanBreak = false;
+ for (const PHINode *Cur : WorkList) {
+ // Don't break PHIs that have no interesting incoming values. That is, where
+ // there is no clear opportunity to fold the "extractelement" instructions
+ // we would add.
+ //
+ // Note: IC does not run after this pass, so we're only interested in the
+ // foldings that the DAG combiner can do.
+ if (any_of(Cur->incoming_values(), isInterestingPHIIncomingValue)) {
+ if (++NumBreakablePHIs >= Threshold) {
+ CanBreak = true;
+ break;
+ }
+ }
}
- if (Score < 0)
- return false;
+ for (const PHINode *Cur : WorkList)
+ BreakPhiNodesCache[Cur] = CanBreak;
- return BreakPhiNodesCache[&I] = true;
+ return CanBreak;
}
/// Helper class for "break large PHIs" (visitPHINode).
@@ -1898,14 +1930,15 @@ bool AMDGPUCodeGenPrepareImpl::visitPHINode(PHINode &I) {
// operations with most elements being "undef". This inhibits a lot of
// optimization opportunities and can result in unreasonably high register
// pressure and the inevitable stack spilling.
- if (!ScalarizeLargePHIs || getCGPassBuilderOption().EnableGlobalISelOption)
+ if (!BreakLargePHIs || getCGPassBuilderOption().EnableGlobalISelOption)
return false;
FixedVectorType *FVT = dyn_cast<FixedVectorType>(I.getType());
- if (!FVT || DL->getTypeSizeInBits(FVT) <= ScalarizeLargePHIsThreshold)
+ if (!FVT || FVT->getNumElements() == 1 ||
+ DL->getTypeSizeInBits(FVT) <= BreakLargePHIsThreshold)
return false;
- if (!ForceScalarizeLargePHIs && !canBreakPHINode(I))
+ if (!ForceBreakLargePHIs && !canBreakPHINode(I))
return false;
std::vector<VectorSlice> Slices;
@@ -1930,8 +1963,7 @@ bool AMDGPUCodeGenPrepareImpl::visitPHINode(PHINode &I) {
Slices.emplace_back(EltTy, Idx, 1);
}
- if (Slices.size() == 1)
- return false;
+ assert(Slices.size() > 1);
// Create one PHI per vector piece. The "VectorSlice" class takes care of
// creating the necessary instruction to extract the relevant slices of each
diff --git a/llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-break-large-phis-heuristics.ll b/llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-break-large-phis-heuristics.ll
index 64d82654a005f9..651b030bd3f79f 100644
--- a/llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-break-large-phis-heuristics.ll
+++ b/llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-break-large-phis-heuristics.ll
@@ -707,6 +707,11 @@ define amdgpu_kernel void @used_by_unbreakable_and_breakable_phi(<5 x double> %i
; CHECK-NEXT: [[LARGEPHI_INSERTSLICE3:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE2]], double [[TMP3]], i64 3
; CHECK-NEXT: [[LARGEPHI_INSERTSLICE4:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE3]], double [[TMP4]], i64 4
; CHECK-NEXT: store <5 x double> [[LARGEPHI_INSERTSLICE4]], ptr [[OUT:%.*]], align 1
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE1:%.*]] = extractelement <5 x double> [[IN]], i64 0
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE3:%.*]] = extractelement <5 x double> [[IN]], i64 1
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE5:%.*]] = extractelement <5 x double> [[IN]], i64 2
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE7:%.*]] = extractelement <5 x double> [[IN]], i64 3
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE9:%.*]] = extractelement <5 x double> [[IN]], i64 4
; CHECK-NEXT: br i1 [[COND2:%.*]], label [[THEN1:%.*]], label [[END:%.*]]
; CHECK: then1:
; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE01:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 0
@@ -714,21 +719,35 @@ define amdgpu_kernel void @used_by_unbreakable_and_breakable_phi(<5 x double> %i
; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE43:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 2
; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE64:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 3
; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE85:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 4
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE011:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 0
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE212:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 1
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE413:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 2
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE614:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 3
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE815:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 4
; CHECK-NEXT: br label [[END]]
; CHECK: end:
-; CHECK-NEXT: [[ENDVAL:%.*]] = phi <5 x double> [ [[LARGEPHI_INSERTSLICE4]], [[THEN1]] ], [ [[IN]], [[FINALLY]] ]
-; CHECK-NEXT: [[TMP5:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE01]], [[THEN1]] ], [ 0.000000e+00, [[FINALLY]] ]
-; CHECK-NEXT: [[TMP6:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE22]], [[THEN1]] ], [ 0.000000e+00, [[FINALLY]] ]
-; CHECK-NEXT: [[TMP7:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE43]], [[THEN1]] ], [ 0.000000e+00, [[FINALLY]] ]
-; CHECK-NEXT: [[TMP8:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE64]], [[THEN1]] ], [ 0.000000e+00, [[FINALLY]] ]
-; CHECK-NEXT: [[TMP9:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE85]], [[THEN1]] ], [ 0.000000e+00, [[FINALLY]] ]
+; CHECK-NEXT: [[TMP5:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE01]], [[THEN1]] ], [ [[LARGEPHI_EXTRACTSLICE1]], [[FINALLY]] ]
+; CHECK-NEXT: [[TMP6:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE22]], [[THEN1]] ], [ [[LARGEPHI_EXTRACTSLICE3]], [[FINALLY]] ]
+; CHECK-NEXT: [[TMP7:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE43]], [[THEN1]] ], [ [[LARGEPHI_EXTRACTSLICE5]], [[FINALLY]] ]
+; CHECK-NEXT: [[TMP8:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE64]], [[THEN1]] ], [ [[LARGEPHI_EXTRACTSLICE7]], [[FINALLY]] ]
+; CHECK-NEXT: [[TMP9:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE85]], [[THEN1]] ], [ [[LARGEPHI_EXTRACTSLICE9]], [[FINALLY]] ]
+; CHECK-NEXT: [[TMP10:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE011]], [[THEN1]] ], [ 0.000000e+00, [[FINALLY]] ]
+; CHECK-NEXT: [[TMP11:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE212]], [[THEN1]] ], [ 0.000000e+00, [[FINALLY]] ]
+; CHECK-NEXT: [[TMP12:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE413]], [[THEN1]] ], [ 0.000000e+00, [[FINALLY]] ]
+; CHECK-NEXT: [[TMP13:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE614]], [[THEN1]] ], [ 0.000000e+00, [[FINALLY]] ]
+; CHECK-NEXT: [[TMP14:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE815]], [[THEN1]] ], [ 0.000000e+00, [[FINALLY]] ]
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE016:%.*]] = insertelement <5 x double> poison, double [[TMP10]], i64 0
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE117:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE016]], double [[TMP11]], i64 1
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE218:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE117]], double [[TMP12]], i64 2
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE319:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE218]], double [[TMP13]], i64 3
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE420:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE319]], double [[TMP14]], i64 4
; CHECK-NEXT: [[LARGEPHI_INSERTSLICE06:%.*]] = insertelement <5 x double> poison, double [[TMP5]], i64 0
; CHECK-NEXT: [[LARGEPHI_INSERTSLICE17:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE06]], double [[TMP6]], i64 1
; CHECK-NEXT: [[LARGEPHI_INSERTSLICE28:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE17]], double [[TMP7]], i64 2
; CHECK-NEXT: [[LARGEPHI_INSERTSLICE39:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE28]], double [[TMP8]], i64 3
; CHECK-NEXT: [[LARGEPHI_INSERTSLICE410:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE39]], double [[TMP9]], i64 4
-; CHECK-NEXT: store <5 x double> [[ENDVAL]], ptr [[OUT]], align 1
; CHECK-NEXT: store <5 x double> [[LARGEPHI_INSERTSLICE410]], ptr [[OUT]], align 1
+; CHECK-NEXT: store <5 x double> [[LARGEPHI_INSERTSLICE420]], ptr [[OUT]], align 1
; CHECK-NEXT: ret void
;
entry:
@@ -784,3 +803,468 @@ loop:
end:
ret void
}
+
+
+define amdgpu_kernel void @test_breakable_chain_2_out_of_4(<5 x double> %in, ptr %out, i1 %cond) {
+; CHECK-LABEL: @test_breakable_chain_2_out_of_4(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: br i1 [[COND:%.*]], label [[COND0_TRUE:%.*]], label [[COND0_END:%.*]]
+; CHECK: cond0.true:
+; CHECK-NEXT: br label [[COND0_END]]
+; CHECK: cond0.end:
+; CHECK-NEXT: [[VAL_0:%.*]] = phi <5 x double> [ zeroinitializer, [[ENTRY:%.*]] ], [ zeroinitializer, [[COND0_TRUE]] ]
+; CHECK-NEXT: br i1 [[COND]], label [[COND1_TRUE:%.*]], label [[COND1_END:%.*]]
+; CHECK: cond1.true:
+; CHECK-NEXT: br label [[COND1_END]]
+; CHECK: cond1.end:
+; CHECK-NEXT: [[VAL_1:%.*]] = phi <5 x double> [ [[VAL_0]], [[COND0_END]] ], [ zeroinitializer, [[COND1_TRUE]] ]
+; CHECK-NEXT: br i1 [[COND]], label [[COND2_TRUE:%.*]], label [[COND2_END:%.*]]
+; CHECK: cond2.true:
+; CHECK-NEXT: br i1 [[COND]], label [[COND2_END]], label [[END:%.*]]
+; CHECK: cond2.end:
+; CHECK-NEXT: [[VAL_2:%.*]] = phi <5 x double> [ [[VAL_1]], [[COND1_END]] ], [ [[IN:%.*]], [[COND2_TRUE]] ]
+; CHECK-NEXT: br label [[END]]
+; CHECK: end:
+; CHECK-NEXT: [[ENDVAL:%.*]] = phi <5 x double> [ [[VAL_2]], [[COND2_END]] ], [ [[IN]], [[COND2_TRUE]] ]
+; CHECK-NEXT: store <5 x double> [[ENDVAL]], ptr [[OUT:%.*]], align 1
+; CHECK-NEXT: ret void
+;
+entry:
+ br i1 %cond, label %cond0.true, label %cond0.end
+
+cond0.true:
+ br label %cond0.end
+
+cond0.end:
+ %val.0 = phi <5 x double> [ zeroinitializer, %entry ], [ zeroinitializer, %cond0.true ]
+ br i1 %cond, label %cond1.true, label %cond1.end
+
+cond1.true:
+ br label %cond1.end
+
+cond1.end:
+ %val.1 = phi <5 x double> [ %val.0, %cond0.end ], [ zeroinitializer, %cond1.true ]
+ br i1 %cond, label %cond2.true, label %cond2.end
+
+cond2.true:
+ br i1 %cond, label %cond2.end, label %end
+
+cond2.end:
+ %val.2 = phi <5 x double> [ %val.1, %cond1.end ], [ %in, %cond2.true ]
+ br label %end
+
+end:
+ %endval = phi <5 x double> [ %val.2, %cond2.end ], [ %in, %cond2.true ]
+ store <5 x double> %endval, ptr %out, align 1
+ ret void
+}
+
+define amdgpu_kernel void @test_breakable_chain_3_out_of_4(<5 x double> %in, ptr %out, i1 %cond) {
+; CHECK-LABEL: @test_breakable_chain_3_out_of_4(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: br i1 [[COND:%.*]], label [[COND0_TRUE:%.*]], label [[COND0_END:%.*]]
+; CHECK: cond0.true:
+; CHECK-NEXT: [[X:%.*]] = insertelement <5 x double> [[IN:%.*]], double 3.140000e+00, i64 3
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE1:%.*]] = extractelement <5 x double> [[IN]], i64 0
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE3:%.*]] = extractelement <5 x double> [[IN]], i64 1
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE5:%.*]] = extractelement <5 x double> [[IN]], i64 2
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE7:%.*]] = extractelement <5 x double> [[IN]], i64 3
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE9:%.*]] = extractelement <5 x double> [[IN]], i64 4
+; CHECK-NEXT: br label [[COND0_END]]
+; CHECK: cond0.end:
+; CHECK-NEXT: [[TMP0:%.*]] = phi double [ 0.000000e+00, [[ENTRY:%.*]] ], [ [[LARGEPHI_EXTRACTSLICE1]], [[COND0_TRUE]] ]
+; CHECK-NEXT: [[TMP1:%.*]] = phi double [ 0.000000e+00, [[ENTRY]] ], [ [[LARGEPHI_EXTRACTSLICE3]], [[COND0_TRUE]] ]
+; CHECK-NEXT: [[TMP2:%.*]] = phi double [ 0.000000e+00, [[ENTRY]] ], [ [[LARGEPHI_EXTRACTSLICE5]], [[COND0_TRUE]] ]
+; CHECK-NEXT: [[TMP3:%.*]] = phi double [ 0.000000e+00, [[ENTRY]] ], [ [[LARGEPHI_EXTRACTSLICE7]], [[COND0_TRUE]] ]
+; CHECK-NEXT: [[TMP4:%.*]] = phi double [ 0.000000e+00, [[ENTRY]] ], [ [[LARGEPHI_EXTRACTSLICE9]], [[COND0_TRUE]] ]
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE0:%.*]] = insertelement <5 x double> poison, double [[TMP0]], i64 0
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE1:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE0]], double [[TMP1]], i64 1
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE2:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE1]], double [[TMP2]], i64 2
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE3:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE2]], double [[TMP3]], i64 3
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE4:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE3]], double [[TMP4]], i64 4
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE0:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 0
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE2:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 1
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE4:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 2
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE6:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 3
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE8:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 4
+; CHECK-NEXT: br i1 [[COND]], label [[COND1_TRUE:%.*]], label [[COND1_END:%.*]]
+; CHECK: cond1.true:
+; CHECK-NEXT: br label [[COND1_END]]
+; CHECK: cond1.end:
+; CHECK-NEXT: [[TMP5:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE0]], [[COND0_END]] ], [ 0.000000e+00, [[COND1_TRUE]] ]
+; CHECK-NEXT: [[TMP6:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE2]], [[COND0_END]] ], [ 0.000000e+00, [[COND1_TRUE]] ]
+; CHECK-NEXT: [[TMP7:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE4]], [[COND0_END]] ], [ 0.000000e+00, [[COND1_TRUE]] ]
+; CHECK-NEXT: [[TMP8:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE6]], [[COND0_END]] ], [ 0.000000e+00, [[COND1_TRUE]] ]
+; CHECK-NEXT: [[TMP9:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE8]], [[COND0_END]] ], [ 0.000000e+00, [[COND1_TRUE]] ]
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE01:%.*]] = insertelement <5 x double> poison, double [[TMP5]], i64 0
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE12:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE01]], double [[TMP6]], i64 1
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE23:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE12]], double [[TMP7]], i64 2
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE34:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE23]], double [[TMP8]], i64 3
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE45:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE34]], double [[TMP9]], i64 4
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE06:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE45]], i64 0
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE27:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE45]], i64 1
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE48:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE45]], i64 2
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE69:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE45]], i64 3
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE810:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE45]], i64 4
+; CHECK-NEXT: br i1 [[COND]], label [[COND2_TRUE:%.*]], label [[COND2_END:%.*]]
+; CHECK: cond2.true:
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE117:%.*]] = extractelement <5 x double> [[IN]], i64 0
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE319:%.*]] = extractelement <5 x double> [[IN]], i64 1
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE521:%.*]] = extractelement <5 x double> [[IN]], i64 2
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE723:%.*]] = extractelement <5 x double> [[IN]], i64 3
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE925:%.*]] = extractelement <5 x double> [[IN]], i64 4
+; CHECK-NEXT: br i1 [[COND]], label [[COND2_END]], label [[END:%.*]]
+; CHECK: cond2.end:
+; CHECK-NEXT: [[TMP10:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE06]], [[COND1_END]] ], [ 0.000000e+00, [[COND2_TRUE]] ]
+; CHECK-NEXT: [[TMP11:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE27]], [[COND1_END]] ], [ 0.000000e+00, [[COND2_TRUE]] ]
+; CHECK-NEXT: [[TMP12:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE48]], [[COND1_END]] ], [ 0.000000e+00, [[COND2_TRUE]] ]
+; CHECK-NEXT: [[TMP13:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE69]], [[COND1_END]] ], [ 0.000000e+00, [[COND2_TRUE]] ]
+; CHECK-NEXT: [[TMP14:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE810]], [[COND1_END]] ], [ 0.000000e+00, [[COND2_TRUE]] ]
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE011:%.*]] = insertelement <5 x double> poison, double [[TMP10]], i64 0
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE112:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE011]], double [[TMP11]], i64 1
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE213:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE112]], double [[TMP12]], i64 2
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE314:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE213]], double [[TMP13]], i64 3
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE415:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE314]], double [[TMP14]], i64 4
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE016:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE415]], i64 0
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE218:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE415]], i64 1
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE420:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE415]], i64 2
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE622:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE415]], i64 3
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE824:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE415]], i64 4
+; CHECK-NEXT: br label [[END]]
+; CHECK: end:
+; CHECK-NEXT: [[TMP15:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE016]], [[COND2_END]] ], [ [[LARGEPHI_EXTRACTSLICE117]], [[COND2_TRUE]] ]
+; CHECK-NEXT: [[TMP16:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE218]], [[COND2_END]] ], [ [[LARGEPHI_EXTRACTSLICE319]], [[COND2_TRUE]] ]
+; CHECK-NEXT: [[TMP17:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE420]], [[COND2_END]] ], [ [[LARGEPHI_EXTRACTSLICE521]], [[COND2_TRUE]] ]
+; CHECK-NEXT: [[TMP18:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE622]], [[COND2_END]] ], [ [[LARGEPHI_EXTRACTSLICE723]], [[COND2_TRUE]] ]
+; CHECK-NEXT: [[TMP19:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE824]], [[COND2_END]] ], [ [[LARGEPHI_EXTRACTSLICE925]], [[COND2_TRUE]] ]
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE026:%.*]] = insertelement <5 x double> poison, double [[TMP15]], i64 0
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE127:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE026]], double [[TMP16]], i64 1
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE228:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE127]], double [[TMP17]], i64 2
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE329:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE228]], double [[TMP18]], i64 3
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE430:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE329]], double [[TMP19]], i64 4
+; CHECK-NEXT: store <5 x double> [[LARGEPHI_INSERTSLICE430]], ptr [[OUT:%.*]], align 1
+; CHECK-NEXT: ret void
+;
+entry:
+ br i1 %cond, label %cond0.true, label %cond0.end
+
+cond0.true:
+ %x = insertelement <5 x double> %in, double 3.140000e+00, i64 3
+ br label %cond0.end
+
+cond0.end:
+ %val.0 = phi <5 x double> [ zeroinitializer, %entry ], [ %in, %cond0.true ]
+ br i1 %cond, label %cond1.true, label %cond1.end
+
+cond1.true:
+ br label %cond1.end
+
+cond1.end:
+ %val.1 = phi <5 x double> [ %val.0, %cond0.end ], [ zeroinitializer, %cond1.true ]
+ br i1 %cond, label %cond2.true, label %cond2.end
+
+cond2.true:
+ br i1 %cond, label %cond2.end, label %end
+
+cond2.end:
+ %val.2 = phi <5 x double> [ %val.1, %cond1.end ], [ zeroinitializer, %cond2.true ]
+ br label %end
+
+end:
+ %endval = phi <5 x double> [ %val.2, %cond2.end ], [ %in, %cond2.true ]
+ store <5 x double> %endval, ptr %out, align 1
+ ret void
+}
+
+; 7 PHIS, 4 Breakable
+define amdgpu_kernel void @test_breakable_chain_4_out_of_7(<5 x double> %in, ptr %out, i1 %cond) {
+; CHECK-LABEL: @test_breakable_chain_4_out_of_7(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: br i1 [[COND:%.*]], label [[COND0_TRUE:%.*]], label [[COND0_END:%.*]]
+; CHECK: cond0.true:
+; CHECK-NEXT: br label [[COND0_END]]
+; CHECK: cond0.end:
+; CHECK-NEXT: [[VAL_0:%.*]] = phi <5 x double> [ zeroinitializer, [[ENTRY:%.*]] ], [ [[IN:%.*]], [[COND0_TRUE]] ]
+; CHECK-NEXT: br i1 [[COND]], label [[COND1_TRUE:%.*]], label [[COND1_END:%.*]]
+; CHECK: cond1.true:
+; CHECK-NEXT: br label [[COND1_END]]
+; CHECK: cond1.end:
+; CHECK-NEXT: [[VAL_1:%.*]] = phi <5 x double> [ [[VAL_0]], [[COND0_END]] ], [ zeroinitializer, [[COND1_TRUE]] ]
+; CHECK-NEXT: br i1 [[COND]], label [[COND2_TRUE:%.*]], label [[COND2_END:%.*]]
+; CHECK: cond2.true:
+; CHECK-NEXT: br label [[COND2_END]]
+; CHECK: cond2.end:
+; CHECK-NEXT: [[VAL_2:%.*]] = phi <5 x double> [ [[VAL_1]], [[COND1_END]] ], [ zeroinitializer, [[COND2_TRUE]] ]
+; CHECK-NEXT: br i1 [[COND]], label [[COND3_TRUE:%.*]], label [[COND3_END:%.*]]
+; CHECK: cond3.true:
+; CHECK-NEXT: br label [[COND3_END]]
+; CHECK: cond3.end:
+; CHECK-NEXT: [[VAL_3:%.*]] = phi <5 x double> [ [[VAL_2]], [[COND2_END]] ], [ zeroinitializer, [[COND3_TRUE]] ]
+; CHECK-NEXT: br i1 [[COND]], label [[COND4_TRUE:%.*]], label [[COND4_END:%.*]]
+; CHECK: cond4.true:
+; CHECK-NEXT: [[B:%.*]] = insertelement <5 x double> [[VAL_0]], double 7.140000e+00, i64 4
+; CHECK-NEXT: br label [[COND4_END]]
+; CHECK: cond4.end:
+; CHECK-NEXT: [[VAL_4:%.*]] = phi <5 x double> [ [[VAL_3]], [[COND3_END]] ], [ [[IN]], [[COND4_TRUE]] ]
+; CHECK-NEXT: br i1 [[COND]], label [[COND5_TRUE:%.*]], label [[COND5_END:%.*]]
+; CHECK: cond5.true:
+; CHECK-NEXT: br i1 [[COND]], label [[END:%.*]], label [[COND5_END]]
+; CHECK: cond5.end:
+; CHECK-NEXT: [[VAL_5:%.*]] = phi <5 x double> [ [[VAL_4]], [[COND4_END]] ], [ [[IN]], [[COND5_TRUE]] ]
+; CHECK-NEXT: br label [[END]]
+; CHECK: end:
+; CHECK-NEXT: [[ENDVAL:%.*]] = phi <5 x double> [ [[VAL_5]], [[COND5_END]] ], [ [[IN]], [[COND5_TRUE]] ]
+; CHECK-NEXT: store <5 x double> [[ENDVAL]], ptr [[OUT:%.*]], align 1
+; CHECK-NEXT: ret void
+;
+entry:
+ br i1 %cond, label %cond0.true, label %cond0.end
+
+cond0.true:
+ br label %cond0.end
+
+cond0.end:
+ %val.0 = phi <5 x double> [ zeroinitializer, %entry ], [ %in, %cond0.true ]
+ br i1 %cond, label %cond1.true, label %cond1.end
+
+cond1.true:
+ br label %cond1.end
+
+cond1.end:
+ %val.1 = phi <5 x double> [ %val.0, %cond0.end ], [ zeroinitializer, %cond1.true ]
+ br i1 %cond, label %cond2.true, label %cond2.end
+
+cond2.true:
+ br label %cond2.end
+
+cond2.end:
+ %val.2 = phi <5 x double> [ %val.1, %cond1.end ], [ zeroinitializer, %cond2.true ]
+ br i1 %cond, label %cond3.true, label %cond3.end
+
+cond3.true:
+ br label %cond3.end
+
+cond3.end:
+ %val.3 = phi <5 x double> [ %val.2, %cond2.end ], [ zeroinitializer, %cond3.true ]
+ br i1 %cond, label %cond4.true, label %cond4.end
+
+cond4.true:
+ %b = insertelement <5 x double> %val.0, double 7.140000e+00, i64 4
+ br label %cond4.end
+
+cond4.end:
+ %val.4 = phi <5 x double> [ %val.3, %cond3.end ], [ %in, %cond4.true ]
+ br i1 %cond, label %cond5.true, label %cond5.end
+
+cond5.true:
+ br i1 %cond, label %end, label %cond5.end
+
+cond5.end:
+ %val.5 = phi <5 x double> [ %val.4, %cond4.end ], [ %in, %cond5.true ]
+ br label %end
+
+end:
+ %endval = phi <5 x double> [ %val.5, %cond5.end ], [ %in, %cond5.true ]
+ store <5 x double> %endval, ptr %out, align 1
+ ret void
+}
+
+define amdgpu_kernel void @test_breakable_chain_5_out_of_7(<5 x double> %in, ptr %out, i1 %cond) {
+; CHECK-LABEL: @test_breakable_chain_5_out_of_7(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE0:%.*]] = extractelement <5 x double> [[IN:%.*]], i64 0
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE2:%.*]] = extractelement <5 x double> [[IN]], i64 1
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE4:%.*]] = extractelement <5 x double> [[IN]], i64 2
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE6:%.*]] = extractelement <5 x double> [[IN]], i64 3
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE8:%.*]] = extractelement <5 x double> [[IN]], i64 4
+; CHECK-NEXT: br i1 [[COND:%.*]], label [[COND0_TRUE:%.*]], label [[COND0_END:%.*]]
+; CHECK: cond0.true:
+; CHECK-NEXT: br label [[COND0_END]]
+; CHECK: cond0.end:
+; CHECK-NEXT: [[TMP0:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE0]], [[ENTRY:%.*]] ], [ 0.000000e+00, [[COND0_TRUE]] ]
+; CHECK-NEXT: [[TMP1:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE2]], [[ENTRY]] ], [ 0.000000e+00, [[COND0_TRUE]] ]
+; CHECK-NEXT: [[TMP2:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE4]], [[ENTRY]] ], [ 0.000000e+00, [[COND0_TRUE]] ]
+; CHECK-NEXT: [[TMP3:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE6]], [[ENTRY]] ], [ 0.000000e+00, [[COND0_TRUE]] ]
+; CHECK-NEXT: [[TMP4:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE8]], [[ENTRY]] ], [ 0.000000e+00, [[COND0_TRUE]] ]
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE0:%.*]] = insertelement <5 x double> poison, double [[TMP0]], i64 0
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE1:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE0]], double [[TMP1]], i64 1
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE2:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE1]], double [[TMP2]], i64 2
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE3:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE2]], double [[TMP3]], i64 3
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE4:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE3]], double [[TMP4]], i64 4
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE01:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 0
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE22:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 1
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE43:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 2
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE64:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 3
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE85:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE4]], i64 4
+; CHECK-NEXT: br i1 [[COND]], label [[COND1_TRUE:%.*]], label [[COND1_END:%.*]]
+; CHECK: cond1.true:
+; CHECK-NEXT: br label [[COND1_END]]
+; CHECK: cond1.end:
+; CHECK-NEXT: [[TMP5:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE01]], [[COND0_END]] ], [ 0.000000e+00, [[COND1_TRUE]] ]
+; CHECK-NEXT: [[TMP6:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE22]], [[COND0_END]] ], [ 0.000000e+00, [[COND1_TRUE]] ]
+; CHECK-NEXT: [[TMP7:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE43]], [[COND0_END]] ], [ 0.000000e+00, [[COND1_TRUE]] ]
+; CHECK-NEXT: [[TMP8:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE64]], [[COND0_END]] ], [ 0.000000e+00, [[COND1_TRUE]] ]
+; CHECK-NEXT: [[TMP9:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE85]], [[COND0_END]] ], [ 0.000000e+00, [[COND1_TRUE]] ]
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE06:%.*]] = insertelement <5 x double> poison, double [[TMP5]], i64 0
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE17:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE06]], double [[TMP6]], i64 1
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE28:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE17]], double [[TMP7]], i64 2
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE39:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE28]], double [[TMP8]], i64 3
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE410:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE39]], double [[TMP9]], i64 4
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE011:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE410]], i64 0
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE212:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE410]], i64 1
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE413:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE410]], i64 2
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE614:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE410]], i64 3
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE815:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE410]], i64 4
+; CHECK-NEXT: br i1 [[COND]], label [[COND2_TRUE:%.*]], label [[COND2_END:%.*]]
+; CHECK: cond2.true:
+; CHECK-NEXT: br label [[COND2_END]]
+; CHECK: cond2.end:
+; CHECK-NEXT: [[TMP10:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE011]], [[COND1_END]] ], [ 0.000000e+00, [[COND2_TRUE]] ]
+; CHECK-NEXT: [[TMP11:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE212]], [[COND1_END]] ], [ 0.000000e+00, [[COND2_TRUE]] ]
+; CHECK-NEXT: [[TMP12:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE413]], [[COND1_END]] ], [ 0.000000e+00, [[COND2_TRUE]] ]
+; CHECK-NEXT: [[TMP13:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE614]], [[COND1_END]] ], [ 0.000000e+00, [[COND2_TRUE]] ]
+; CHECK-NEXT: [[TMP14:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE815]], [[COND1_END]] ], [ 0.000000e+00, [[COND2_TRUE]] ]
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE016:%.*]] = insertelement <5 x double> poison, double [[TMP10]], i64 0
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE117:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE016]], double [[TMP11]], i64 1
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE218:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE117]], double [[TMP12]], i64 2
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE319:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE218]], double [[TMP13]], i64 3
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE420:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE319]], double [[TMP14]], i64 4
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE021:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE420]], i64 0
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE222:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE420]], i64 1
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE423:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE420]], i64 2
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE624:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE420]], i64 3
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE825:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE420]], i64 4
+; CHECK-NEXT: br i1 [[COND]], label [[COND3_TRUE:%.*]], label [[COND3_END:%.*]]
+; CHECK: cond3.true:
+; CHECK-NEXT: [[A:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE4]], double 7.140000e+00, i64 4
+; CHECK-NEXT: br label [[COND3_END]]
+; CHECK: cond3.end:
+; CHECK-NEXT: [[TMP15:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE021]], [[COND2_END]] ], [ 0.000000e+00, [[COND3_TRUE]] ]
+; CHECK-NEXT: [[TMP16:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE222]], [[COND2_END]] ], [ 0.000000e+00, [[COND3_TRUE]] ]
+; CHECK-NEXT: [[TMP17:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE423]], [[COND2_END]] ], [ 0.000000e+00, [[COND3_TRUE]] ]
+; CHECK-NEXT: [[TMP18:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE624]], [[COND2_END]] ], [ 0.000000e+00, [[COND3_TRUE]] ]
+; CHECK-NEXT: [[TMP19:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE825]], [[COND2_END]] ], [ 0.000000e+00, [[COND3_TRUE]] ]
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE026:%.*]] = insertelement <5 x double> poison, double [[TMP15]], i64 0
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE127:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE026]], double [[TMP16]], i64 1
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE228:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE127]], double [[TMP17]], i64 2
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE329:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE228]], double [[TMP18]], i64 3
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE430:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE329]], double [[TMP19]], i64 4
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE031:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE430]], i64 0
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE232:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE430]], i64 1
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE433:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE430]], i64 2
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE634:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE430]], i64 3
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE835:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE430]], i64 4
+; CHECK-NEXT: br i1 [[COND]], label [[COND4_TRUE:%.*]], label [[COND4_END:%.*]]
+; CHECK: cond4.true:
+; CHECK-NEXT: br label [[COND4_END]]
+; CHECK: cond4.end:
+; CHECK-NEXT: [[TMP20:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE031]], [[COND3_END]] ], [ 0.000000e+00, [[COND4_TRUE]] ]
+; CHECK-NEXT: [[TMP21:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE232]], [[COND3_END]] ], [ 0.000000e+00, [[COND4_TRUE]] ]
+; CHECK-NEXT: [[TMP22:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE433]], [[COND3_END]] ], [ 0.000000e+00, [[COND4_TRUE]] ]
+; CHECK-NEXT: [[TMP23:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE634]], [[COND3_END]] ], [ 0.000000e+00, [[COND4_TRUE]] ]
+; CHECK-NEXT: [[TMP24:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE835]], [[COND3_END]] ], [ 0.000000e+00, [[COND4_TRUE]] ]
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE036:%.*]] = insertelement <5 x double> poison, double [[TMP20]], i64 0
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE137:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE036]], double [[TMP21]], i64 1
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE238:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE137]], double [[TMP22]], i64 2
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE339:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE238]], double [[TMP23]], i64 3
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE440:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE339]], double [[TMP24]], i64 4
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE041:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE440]], i64 0
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE242:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE440]], i64 1
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE443:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE440]], i64 2
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE644:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE440]], i64 3
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE845:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE440]], i64 4
+; CHECK-NEXT: br i1 [[COND]], label [[COND5_TRUE:%.*]], label [[COND5_END:%.*]]
+; CHECK: cond5.true:
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE1:%.*]] = extractelement <5 x double> [[IN]], i64 0
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE3:%.*]] = extractelement <5 x double> [[IN]], i64 1
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE5:%.*]] = extractelement <5 x double> [[IN]], i64 2
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE7:%.*]] = extractelement <5 x double> [[IN]], i64 3
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE9:%.*]] = extractelement <5 x double> [[IN]], i64 4
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE152:%.*]] = extractelement <5 x double> [[IN]], i64 0
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE354:%.*]] = extractelement <5 x double> [[IN]], i64 1
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE556:%.*]] = extractelement <5 x double> [[IN]], i64 2
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE758:%.*]] = extractelement <5 x double> [[IN]], i64 3
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE960:%.*]] = extractelement <5 x double> [[IN]], i64 4
+; CHECK-NEXT: br i1 [[COND]], label [[END:%.*]], label [[COND5_END]]
+; CHECK: cond5.end:
+; CHECK-NEXT: [[TMP25:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE041]], [[COND4_END]] ], [ [[LARGEPHI_EXTRACTSLICE1]], [[COND5_TRUE]] ]
+; CHECK-NEXT: [[TMP26:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE242]], [[COND4_END]] ], [ [[LARGEPHI_EXTRACTSLICE3]], [[COND5_TRUE]] ]
+; CHECK-NEXT: [[TMP27:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE443]], [[COND4_END]] ], [ [[LARGEPHI_EXTRACTSLICE5]], [[COND5_TRUE]] ]
+; CHECK-NEXT: [[TMP28:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE644]], [[COND4_END]] ], [ [[LARGEPHI_EXTRACTSLICE7]], [[COND5_TRUE]] ]
+; CHECK-NEXT: [[TMP29:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE845]], [[COND4_END]] ], [ [[LARGEPHI_EXTRACTSLICE9]], [[COND5_TRUE]] ]
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE046:%.*]] = insertelement <5 x double> poison, double [[TMP25]], i64 0
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE147:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE046]], double [[TMP26]], i64 1
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE248:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE147]], double [[TMP27]], i64 2
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE349:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE248]], double [[TMP28]], i64 3
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE450:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE349]], double [[TMP29]], i64 4
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE051:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE450]], i64 0
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE253:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE450]], i64 1
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE455:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE450]], i64 2
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE657:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE450]], i64 3
+; CHECK-NEXT: [[LARGEPHI_EXTRACTSLICE859:%.*]] = extractelement <5 x double> [[LARGEPHI_INSERTSLICE450]], i64 4
+; CHECK-NEXT: br label [[END]]
+; CHECK: end:
+; CHECK-NEXT: [[TMP30:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE051]], [[COND5_END]] ], [ [[LARGEPHI_EXTRACTSLICE152]], [[COND5_TRUE]] ]
+; CHECK-NEXT: [[TMP31:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE253]], [[COND5_END]] ], [ [[LARGEPHI_EXTRACTSLICE354]], [[COND5_TRUE]] ]
+; CHECK-NEXT: [[TMP32:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE455]], [[COND5_END]] ], [ [[LARGEPHI_EXTRACTSLICE556]], [[COND5_TRUE]] ]
+; CHECK-NEXT: [[TMP33:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE657]], [[COND5_END]] ], [ [[LARGEPHI_EXTRACTSLICE758]], [[COND5_TRUE]] ]
+; CHECK-NEXT: [[TMP34:%.*]] = phi double [ [[LARGEPHI_EXTRACTSLICE859]], [[COND5_END]] ], [ [[LARGEPHI_EXTRACTSLICE960]], [[COND5_TRUE]] ]
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE061:%.*]] = insertelement <5 x double> poison, double [[TMP30]], i64 0
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE162:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE061]], double [[TMP31]], i64 1
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE263:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE162]], double [[TMP32]], i64 2
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE364:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE263]], double [[TMP33]], i64 3
+; CHECK-NEXT: [[LARGEPHI_INSERTSLICE465:%.*]] = insertelement <5 x double> [[LARGEPHI_INSERTSLICE364]], double [[TMP34]], i64 4
+; CHECK-NEXT: store <5 x double> [[LARGEPHI_INSERTSLICE465]], ptr [[OUT:%.*]], align 1
+; CHECK-NEXT: ret void
+;
+entry:
+ br i1 %cond, label %cond0.true, label %cond0.end
+
+cond0.true:
+ br label %cond0.end
+
+cond0.end:
+ %val.0 = phi <5 x double> [ %in, %entry ], [ zeroinitializer, %cond0.true ]
+ br i1 %cond, label %cond1.true, label %cond1.end
+
+cond1.true:
+ br label %cond1.end
+
+cond1.end:
+ %val.1 = phi <5 x double> [ %val.0, %cond0.end ], [ zeroinitializer, %cond1.true ]
+ br i1 %cond, label %cond2.true, label %cond2.end
+
+cond2.true:
+ br label %cond2.end
+
+cond2.end:
+ %val.2 = phi <5 x double> [ %val.1, %cond1.end ], [ zeroinitializer, %cond2.true ]
+ br i1 %cond, label %cond3.true, label %cond3.end
+
+cond3.true:
+ %a = insertelement <5 x double> %val.0, double 7.140000e+00, i64 4
+ br label %cond3.end
+
+cond3.end:
+ %val.3 = phi <5 x double> [ %val.2, %cond2.end ], [ zeroinitializer, %cond3.true ]
+ br i1 %cond, label %cond4.true, label %cond4.end
+
+cond4.true:
+ br label %cond4.end
+
+cond4.end:
+ %val.4 = phi <5 x double> [ %val.3, %cond3.end ], [ zeroinitializer, %cond4.true ]
+ br i1 %cond, label %cond5.true, label %cond5.end
+
+cond5.true:
+ br i1 %cond, label %end, label %cond5.end
+
+cond5.end:
+ %val.5 = phi <5 x double> [ %val.4, %cond4.end ], [ %in, %cond5.true ]
+ br label %end
+
+end:
+ %endval = phi <5 x double> [ %val.5, %cond5.end ], [ %in, %cond5.true ]
+ store <5 x double> %endval, ptr %out, align 1
+ ret void
+}
More information about the llvm-commits
mailing list