[llvm] [SimplifyCFG][ConstantMerge] Test non-table cases ahead of a switch's lookup table (PR #223255)
Nazar Mokrynskyi via llvm-commits
llvm-commits at lists.llvm.org
Sun Sep 13 09:37:20 PDT 2026
https://github.com/nazar-pc created https://github.com/llvm/llvm-project/pull/223255
Two commits. The first is the change this is about; the second is what stops it from costing binary size.
Fixes #222292.
Despite LLM wrote most of it under my guidance, I edited, audited, tested and benchmarked the changes to the best of my ability before opening the PR.
---
## The problem
`simplifySwitchLookup()` turns a switch whose cases feed constants into phis in one common successor into a lookup table. If a single case does not fit that shape (because it reaches a destination of its own rather than the one the others share), the whole switch is given up and every case pays for the one that did not fit.
Profile-guided builds create exactly that shape. Given a dispatch that loads a handler out of a switch and calls it, indirect call promotion rewrites the call as
if (h == @handler_7) handler_7(...) else h(...)
On the path where the switch took case 7, `h` is already known to be `@handler_7`: the switch answered that question. Jump threading sees this and rewires case 7 straight to the promoted call, deleting the test. Case 7 now reaches a destination of its own and contributes no table entry, so the table is refused. By then inlining has usually replicated the dispatch into every handler, so the loss is paid once per copy, which is how a profile makes a threaded interpreter slower than no profile at all.
## The change
Set those cases aside and test them ahead of the table, leaving the rest of the switch in the shape a table can replace:
if (x == guarded) goto guarded_dest;
... lookup table for the remaining cases ...
A guarded case inside the range the table spans keeps its slot, holding `poison`. Nothing can index the table with a guarded value (the guard is reached first), so the slot needs no bitmask check, and no value has to be invented for it. Guarded cases are also excluded from the counts that decide whether a table is worth building.
Each guard is a test that every value reaching the table runs first, so more than one has to be paid for. Without branch weights only one case is guarded (the single one indirect call promotion leaves behind), with nothing to say it is hot. Note that this one case is guarded whatever its weight, including zero: there is no table without it either way. With weights, up to `-simplifycfg-max-guarded-cases` (default 3) are guarded, and only those taken at least as often as the average case that does reach the table (an integer average, so ties go to guarding).
This matters more than it sounds: indirect call promotion promotes up to three targets per site, and guarding only one leaves the switch without a table anyway. On the workload below nothing promotes a third target, so two is where the benefit was.
Behavior is unchanged where no case is set aside. The common destination is still pinned on the first case that yields results, the majority is consulted only once a case is actually being guarded. Running `opt` over every file in `test/Transforms/SimplifyCFG` with `-simplifycfg-max-guarded-cases=0` produces byte-identical output to the parent commit for all 316 of them, and a separately built unpatched compiler produced a binary matching the guard-off one on the workload below.
## Why the second commit
Inlining replicates a dispatch, and each copy guards a different set of cases, so each copy's table holds poison in different slots. Identical everywhere else, they cannot be merged, and each costs a table and its relocations. On the workload below that was 50 extra 445-entry tables: +178 KB of `.data.rel.ro` and +532 KB of `.rela.dyn`.
`poison` may be replaced by any value, so two arrays that agree wherever both are defined describe the same thing, and unifying them into the more defined of the two refines both. That needs no reasoning about which slots are read, so `ConstantMerge` can do it: a second phase compares poison-holding arrays against the others and merges them. The 50 tables collapse back into one.
That phase cannot use the hash map the existing merging relies on. Two constants that unify need not be equal, and no hash of one finds the other, since they disagree exactly where a hash would read. Only a constant holding poison has anything to gain, so those drive the search, each compared against the constants of its type. It merges into the one needing the fewest elements filled in, so the result does not depend on the order the globals appear in. The search is bounded by `-constmerge-max-poison-candidates` (1024), and giving that up costs an optimization, never correctness.
## Measurements
A RISC-V interpreter in Rust: a `match` over the opcode yields a handler function pointer, then a tail call, with the dispatch inlined into all 448 handlers (supports vector extensions). Without a profile they share one 445-entry table. Rustc was built from source against patched and unpatched LLVM, so the only difference between columns is these commits.
| | before, no PGO | before, PGO | first commit, PGO | both commits, PGO |
|---|---|---|---|---|
| `ed25519_verify` | 728.1 us | 770.4 us | **583.2 us** | 597.2 us |
| `blake3_hash_chunk` | 16.05 us | 20.17 us | 13.98 us | **13.78 us** |
| `.text` | 3,173,358 | 3,230,190 | 2,779,246 | 2,785,518 |
| `.rodata` | 248,564 | 404,020 | 252,532 | 251,764 |
| `.data.rel.ro` | 55,384 | 57,352 | 235,360 | 60,928 |
| `.rela.dyn` | 101,016 | 105,048 | 636,960 | 115,776 |
| total | 3,578,322 | 3,796,610 | 3,904,098 | **3,213,986** |
With this PR, PGO finally produces both smaller and faster binaries.
`ed25519_verify` represents workload with a wider set of instructions (auto-vectorized), while `blake3_hash_chunk` is narrower (didn't auto-vectorize well).
The first commit on its own is a 2.8% size regression: 50 copies of a 445-entry table, each identical to the rest apart from the handful of slots its guards left as `poison`, is +178 KB of `.data.rel.ro` and +532 KB of `.rela.dyn`.
And the second commit recovers 690 KB of size with a small time cost: merging costs 2.4% on `ed25519_verify` while gaining 1.4% on `blake3_hash_chunk`.
It seemed like a good tradeoff overall to have both.
>From d154f404fbe1b2530acfe98dabf2f28307102ce4 Mon Sep 17 00:00:00 2001
From: Nazar Mokrynskyi <nazar at mokrynskyi.com>
Date: Sun, 13 Sep 2026 14:07:47 +0000
Subject: [PATCH 1/2] [SimplifyCFG] Test non-table cases ahead of a switch's
lookup table
A switch that would become a lookup table can be left with a case that
cannot contribute a table entry, because it reaches a destination of its
own rather than the one the other cases share. Indirect call promotion
creates exactly that shape: it gives a profiled target of a dispatch a
destination of its own, and jump threading then folds the test it
inserted back into the switch, since the switch had already established
the value the test asks about. The table is given up for the whole
switch, and every copy the dispatch was inlined into pays for it.
Set those cases aside and test them ahead of the table instead, leaving
the rest of the switch in the shape a table can replace. Each keeps a
slot in the table, holding poison: nothing can index the table with a
guarded value, since the guard is reached first, so the slot needs no
bitmask check and no value has to be invented for it. Guarded cases are
excluded from the counts that decide whether a table is worth building.
Each guarded case is a test that every value reaching the table runs
first, so more than one has to be paid for. Without branch weights only
the single case indirect call promotion leaves behind is guarded; with
them, up to -simplifycfg-max-guarded-cases cases are, and only those
taken at least as often as the average case that reaches the table.
Behaviour is unchanged where no case is set aside: the common
destination is still pinned on the first case, and the majority is only
consulted once a case is being guarded. Setting
-simplifycfg-max-guarded-cases=0 gives up the table instead, as before.
---
llvm/lib/Transforms/Utils/SimplifyCFG.cpp | 275 ++++-
.../SimplifyCFG/X86/switch_to_lookup_table.ll | 25 +-
.../switch-guard-non-table-case.ll | 961 ++++++++++++++++++
3 files changed, 1218 insertions(+), 43 deletions(-)
create mode 100644 llvm/test/Transforms/SimplifyCFG/switch-guard-non-table-case.ll
diff --git a/llvm/lib/Transforms/Utils/SimplifyCFG.cpp b/llvm/lib/Transforms/Utils/SimplifyCFG.cpp
index ca96f2e70d810..458465e2a1401 100644
--- a/llvm/lib/Transforms/Utils/SimplifyCFG.cpp
+++ b/llvm/lib/Transforms/Utils/SimplifyCFG.cpp
@@ -200,6 +200,11 @@ static cl::opt<unsigned> MaxSwitchCasesPerResult(
"max-switch-cases-per-result", cl::Hidden, cl::init(16),
cl::desc("Limit cases to analyze when converting a switch to select"));
+static cl::opt<unsigned> MaxGuardedCases(
+ "simplifycfg-max-guarded-cases", cl::Hidden, cl::init(3),
+ cl::desc("Most cases a switch may have tested ahead of its lookup table; "
+ "zero gives up the table instead, as before"));
+
static cl::opt<unsigned> MaxJumpThreadingLiveBlocks(
"max-jump-threading-live-blocks", cl::Hidden, cl::init(24),
cl::desc("Limit number of blocks a define in a threaded block is allowed "
@@ -217,6 +222,9 @@ STATISTIC(NumLookupTables,
STATISTIC(
NumLookupTablesHoles,
"Number of switch instructions turned into lookup tables (holes checked)");
+STATISTIC(NumLookupTablesGuardedCase,
+ "Number of switch lookup tables that needed a case tested ahead "
+ "of them");
STATISTIC(NumTableCmpReuses, "Number of reused switch table lookup compares");
STATISTIC(NumFoldValueComparisonIntoPredecessors,
"Number of value comparisons folded into predecessor basic blocks");
@@ -6627,15 +6635,16 @@ getCaseResults(SwitchInst *SI, ConstantInt *CaseVal, BasicBlock *CaseDest,
}
}
- // If we did not have a CommonDest before, use the current one.
- if (!*CommonDest)
- *CommonDest = CaseDest;
+ // Hold off on adopting this case's destination as the common one until the
+ // case is known to be usable, so that a case which fails below leaves the
+ // caller no worse off than if it had never been looked at.
+ BasicBlock *NewCommonDest = *CommonDest ? *CommonDest : CaseDest;
// If the destination isn't the common one, abort.
- if (CaseDest != *CommonDest)
+ if (CaseDest != NewCommonDest)
return false;
// Get the values for this case from phi nodes in the destination block.
- for (PHINode &PHI : (*CommonDest)->phis()) {
+ for (PHINode &PHI : NewCommonDest->phis()) {
int Idx = PHI.getBasicBlockIndex(Pred);
if (Idx == -1)
continue;
@@ -6652,7 +6661,11 @@ getCaseResults(SwitchInst *SI, ConstantInt *CaseVal, BasicBlock *CaseDest,
Res.push_back(std::make_pair(&PHI, ConstVal));
}
- return Res.size() > 0;
+ if (Res.empty())
+ return false;
+
+ *CommonDest = NewCommonDest;
+ return true;
}
// Helper function used to add CaseVal to the list of cases that generate
@@ -7366,11 +7379,12 @@ getDenseSwitchRangeReductionShift(ArrayRef<int64_t> Values, int64_t Base,
// TODO: We could support larger than legal types by limiting based on the
// number of loads required and/or table size. If the constants are small we
// could use smaller table entries and extend after the load.
-static bool shouldBuildLookupTable(SwitchInst *SI, uint64_t TableSize,
+static bool shouldBuildLookupTable(SwitchInst *SI, uint64_t NumCases,
+ uint64_t TableSize,
const TargetTransformInfo &TTI,
const DataLayout &DL,
const SmallVector<Type *> &ResultTypes) {
- if (SI->getNumCases() > TableSize)
+ if (NumCases > TableSize)
return false; // TableSize overflowed.
bool AllTablesFitInRegister = true;
@@ -7399,8 +7413,7 @@ static bool shouldBuildLookupTable(SwitchInst *SI, uint64_t TableSize,
if (HasIllegalType)
return false;
- return isSwitchDense(SI->getNumCases(), TableSize,
- SI->getFunction()->hasOptSize());
+ return isSwitchDense(NumCases, TableSize, SI->getFunction()->hasOptSize());
}
static bool shouldUseSwitchConditionAsTableIndex(
@@ -7529,45 +7542,179 @@ static bool simplifySwitchLookup(SwitchInst *SI, IRBuilder<> &Builder,
// common destination, as well as the min and max case values.
assert(!SI->cases().empty());
SwitchInst::CaseIt CI = SI->case_begin();
- ConstantInt *MinCaseVal = CI->getCaseValue();
- ConstantInt *MaxCaseVal = CI->getCaseValue();
+ // Over the cases that make it into the table, so that a guarded case far away
+ // from the others does not stretch the table to cover a range nothing reads.
+ ConstantInt *MinCaseVal = nullptr;
+ ConstantInt *MaxCaseVal = nullptr;
BasicBlock *CommonDest = nullptr;
using ResultListTy = SmallVector<std::pair<ConstantInt *, Constant *>, 4>;
SmallDenseMap<PHINode *, ResultListTy> ResultLists;
+ // Cases a table cannot hold, to be tested ahead of it.
+ struct GuardedCase {
+ ConstantInt *Val;
+ BasicBlock *Dest;
+ bool HasSlot = false;
+ };
+ SmallVector<GuardedCase, 4> Guarded;
+
SmallDenseMap<PHINode *, Constant *> DefaultResults;
SmallVector<Type *> ResultTypes;
SmallVector<PHINode *, 4> PHIs;
+ // Resolve each case on its own first. Letting the first case that resolves
+ // decide the common destination would make the transform depend on the order
+ // the cases happen to be written in, since the case left over is then
+ // whichever one disagrees with that choice rather than the odd one out.
+ using ResultsTy = SmallVector<std::pair<PHINode *, Constant *>, 4>;
+ struct ResolvedCase {
+ ConstantInt *Val;
+ BasicBlock *Succ;
+ BasicBlock *Dest; // Null if the case has no constant to contribute.
+ ResultsTy Results;
+ };
+ SmallVector<ResolvedCase> Cases;
+ SmallMapVector<BasicBlock *, unsigned, 8> DestCounts;
for (SwitchInst::CaseIt E = SI->case_end(); CI != E; ++CI) {
- ConstantInt *CaseVal = CI->getCaseValue();
- if (CaseVal->getValue().slt(MinCaseVal->getValue()))
- MinCaseVal = CaseVal;
- if (CaseVal->getValue().sgt(MaxCaseVal->getValue()))
- MaxCaseVal = CaseVal;
+ ResolvedCase RC{CI->getCaseValue(), CI->getCaseSuccessor(), nullptr, {}};
+ BasicBlock *Dest = nullptr;
+ if (getCaseResults(SI, RC.Val, RC.Succ, &Dest, RC.Results, DL, TTI)) {
+ RC.Dest = Dest;
+ ++DestCounts[Dest];
+ } else {
+ RC.Results.clear();
+ }
+ Cases.push_back(std::move(RC));
+ }
- // Resulting value at phi nodes for this case value.
- using ResultsTy = SmallVector<std::pair<PHINode *, Constant *>, 4>;
- ResultsTy Results;
- if (!getCaseResults(SI, CaseVal, CI->getCaseSuccessor(), &CommonDest,
- Results, DL, TTI))
- return false;
+ // The destination most of the cases agree on is the one a table can stand in
+ // for. Where nothing is guarded they all agree, so this is the destination
+ // the first case reaches, exactly as before. Where one is, taking the
+ // majority keeps an odd first case from pinning a destination the rest
+ // disagree with and rejecting a switch that is nearly all table.
+ unsigned BestCount = 0;
+ for (const auto &[Dest, Count] : DestCounts)
+ if (Count > BestCount) {
+ BestCount = Count;
+ CommonDest = Dest;
+ }
+ if (!CommonDest)
+ return false;
+
+ for (const ResolvedCase &RC : Cases) {
+ if (RC.Dest != CommonDest) {
+ // This case does not reach the common destination with a constant, so a
+ // table cannot stand in for it: it has to go somewhere else and do
+ // something else. Set it aside rather than give up the table. If it is
+ // the only such case it is tested ahead of the table, which leaves the
+ // rest of the switch in the shape a table can replace. Indirect call
+ // promotion creates exactly this shape, by giving the profiled target of
+ // a dispatch a destination of its own.
+ if (Guarded.size() >= (size_t)MaxGuardedCases)
+ return false;
+ // A case that cannot be reached is not worth a test of its own, and
+ // SimplifyCFG removes it in its own time, leaving the hole the table
+ // already knows how to fill.
+ if (isa<UnreachableInst>(RC.Succ->getFirstNonPHIIt()))
+ return false;
+ Guarded.push_back({RC.Val, RC.Succ});
+ continue;
+ }
+
+ if (!MinCaseVal || RC.Val->getValue().slt(MinCaseVal->getValue()))
+ MinCaseVal = RC.Val;
+ if (!MaxCaseVal || RC.Val->getValue().sgt(MaxCaseVal->getValue()))
+ MaxCaseVal = RC.Val;
// Append the result and result types from this case to the list for each
// phi.
- for (const auto &I : Results) {
+ for (const auto &I : RC.Results) {
PHINode *PHI = I.first;
Constant *Value = I.second;
auto [It, Inserted] = ResultLists.try_emplace(PHI);
if (Inserted)
PHIs.push_back(PHI);
- It->second.push_back(std::make_pair(CaseVal, Value));
+ It->second.push_back(std::make_pair(RC.Val, Value));
ResultTypes.push_back(PHI->getType());
}
}
+ // A guarded case inside the range the others span owns a slot in the table.
+ // One outside it does not, and needs nothing done for it.
+ for (GuardedCase &G : Guarded)
+ G.HasSlot = MinCaseVal && G.Val->getValue().sgt(MinCaseVal->getValue()) &&
+ G.Val->getValue().slt(MaxCaseVal->getValue());
+
+ // Decide whether the case set aside above can really be guarded. Nothing has
+ // been mutated yet, so this can still give up on the whole transform.
+ SmallDenseMap<ConstantInt *, uint64_t> CaseWeights;
+ uint64_t DefaultWeight = 0;
+ bool HaveWeights = false;
+ bool WeightsAreExpected = false;
+ if (!Guarded.empty()) {
+ // The table is the point of the guard, so there has to be one worth
+ // building without the guarded cases. Everything else is left to the
+ // decisions the table already has to pass below.
+ if (SI->getNumCases() - Guarded.size() < 3)
+ return false;
+
+ SmallVector<uint32_t> Weights;
+ HaveWeights = extractBranchWeights(*SI, Weights) &&
+ Weights.size() == SI->getNumSuccessors();
+ if (HaveWeights) {
+ WeightsAreExpected = hasBranchWeightOrigin(*SI);
+ DefaultWeight = Weights[0];
+ for (const auto &Case : SI->cases())
+ CaseWeights[Case.getCaseValue()] = Weights[Case.getSuccessorIndex()];
+ }
+
+ // Each guarded case is a test every value that does reach the table has to
+ // run first, so more than one has to be paid for. Without a profile there
+ // is nothing to say the case is taken often enough to be worth the test, so
+ // only the single case indirect call promotion leaves behind is guarded.
+ // With one, a case earns its test by being taken more often than the
+ // average case that reaches the table.
+ if (Guarded.size() > 1) {
+ if (!HaveWeights)
+ return false;
+ uint64_t TableWeight = 0, TableCases = 0;
+ for (const auto &Case : SI->cases())
+ if (none_of(Guarded, [&](const GuardedCase &G) {
+ return G.Val == Case.getCaseValue();
+ })) {
+ TableWeight += CaseWeights.lookup(Case.getCaseValue());
+ ++TableCases;
+ }
+ uint64_t Average = TableCases ? TableWeight / TableCases : 0;
+ for (const GuardedCase &G : Guarded)
+ if (CaseWeights.lookup(G.Val) < Average)
+ return false;
+ }
+
+ // Keep the rewrite simple by requiring the two halves to be disjoint, so no
+ // phi ends up with edges from both switches.
+ for (const GuardedCase &G : Guarded) {
+ if (G.Dest == SI->getDefaultDest())
+ return false;
+ for (const auto &Case : SI->cases())
+ if (none_of(Guarded,
+ [&](const GuardedCase &O) {
+ return O.Val == Case.getCaseValue();
+ }) &&
+ Case.getCaseSuccessor() == G.Dest)
+ return false;
+ }
+ }
+
+ assert(MinCaseVal && MaxCaseVal && "No case reaches the common destination?");
+
+ // The guarded case never reaches the table, so it does not count towards the
+ // table's density, its size, or the case count that decides whether a hole
+ // check is worth paying for.
+ uint64_t NumTableCases = SI->getNumCases() - Guarded.size();
+
// If the table has holes, we need a constant result for the default case
// or a bitmask that fits in a register.
SmallVector<std::pair<PHINode *, Constant *>, 4> DefaultResultsList;
@@ -7580,6 +7727,22 @@ static bool simplifySwitchLookup(SwitchInst *SI, IRBuilder<> &Builder,
DefaultResults[PHI] = Result;
}
+ // Give each guarded slot poison. Nothing can index the table with a guarded
+ // value, since the guard is reached first, so the slot is unreachable rather
+ // than a hole: it needs no bitmask check, and no value has to be invented for
+ // it. Copies of a dispatch guard different cases, so it also leaves their
+ // tables agreeing wherever both are defined, which is what lets ConstantMerge
+ // fold them back into one.
+ for (const GuardedCase &G : Guarded) {
+ if (!G.HasSlot)
+ continue;
+ for (PHINode *PHI : PHIs) {
+ ResultListTy &ResultList = ResultLists[PHI];
+ Type *ResultType = ResultList.front().second->getType();
+ ResultList.emplace_back(G.Val, PoisonValue::get(ResultType));
+ }
+ }
+
bool UseSwitchConditionAsTableIndex = shouldUseSwitchConditionAsTableIndex(
*MinCaseVal, *MaxCaseVal, HasDefaultResults, ResultTypes, DL, TTI);
uint64_t TableSize;
@@ -7616,13 +7779,14 @@ static bool simplifySwitchLookup(SwitchInst *SI, IRBuilder<> &Builder,
bool NeedMask = AllHolesArePoison && DefaultIsReachable;
if (NeedMask) {
// As an extra penalty for the validity test we require more cases.
- if (SI->getNumCases() < 4) // FIXME: Find best threshold value (benchmark).
+ if (NumTableCases < 4) // FIXME: Find best threshold value (benchmark).
return false;
if (!DL.fitsInLegalInteger(TableSize))
return false;
}
- if (!shouldBuildLookupTable(SI, TableSize, TTI, DL, ResultTypes))
+ if (!shouldBuildLookupTable(SI, NumTableCases, TableSize, TTI, DL,
+ ResultTypes))
return false;
// Compute the table index value.
@@ -7689,9 +7853,64 @@ static bool simplifySwitchLookup(SwitchInst *SI, IRBuilder<> &Builder,
// In the early optimization pipeline, disable formation of lookup tables,
// bit maps and mask checks, as they may inhibit further optimization.
if (!ConvertSwitchToLookupTable &&
- (AnyLookupTables || AnyBitMaps || NeedMask))
+ (AnyLookupTables || AnyBitMaps || NeedMask || !Guarded.empty()))
return false;
+ // From here on the function is mutated. Move the switch into a block of its
+ // own and put a switch over the guarded cases in front of it, so that what is
+ // left below is an ordinary switch whose cases all reach CommonDest.
+ if (!Guarded.empty()) {
+ BasicBlock *TableBB =
+ SplitBlock(BB, SI->getIterator(), DTU, /*LI=*/nullptr,
+ /*MSSAU=*/nullptr, BB->getName() + ".lookup");
+ // The split has already pointed the successors' phis at TableBB.
+ for (const GuardedCase &G : Guarded)
+ SI->removeCase(SI->findCaseValue(G.Val));
+
+ SmallVector<uint64_t> KeptWeights{DefaultWeight};
+ uint64_t ToTableWeight = DefaultWeight;
+ for (const auto &Case : SI->cases()) {
+ uint64_t W = CaseWeights.lookup(Case.getCaseValue());
+ KeptWeights.push_back(W);
+ ToTableWeight += W;
+ }
+ // Only when the switch had a profile to divide up. A switch without one
+ // must not come out of this carrying a made-up profile of zeroes.
+ if (HaveWeights)
+ setFittedBranchWeights(*SI, KeptWeights, WeightsAreExpected);
+
+ BB->getTerminator()->eraseFromParent();
+ SwitchInst *GuardSI =
+ SwitchInst::Create(SI->getCondition(), TableBB, Guarded.size(), BB);
+ SmallVector<uint64_t> GuardWeights{ToTableWeight};
+ for (const GuardedCase &G : Guarded) {
+ GuardSI->addCase(G.Val, G.Dest);
+ GuardWeights.push_back(CaseWeights.lookup(G.Val));
+ }
+ if (HaveWeights)
+ setFittedBranchWeights(*GuardSI, GuardWeights, WeightsAreExpected);
+ GuardSI->setDebugLoc(SI->getDebugLoc());
+
+ // The guarded destinations are reached from BB again rather than from the
+ // block the split put the switch in. Two guarded cases may share one, so
+ // rewrite each destination only once.
+ SmallVector<DominatorTree::UpdateType, 8> Updates;
+ SmallPtrSet<BasicBlock *, 4> Rewritten;
+ for (const GuardedCase &G : Guarded)
+ if (Rewritten.insert(G.Dest).second) {
+ G.Dest->replacePhiUsesWith(TableBB, BB);
+ Updates.push_back({DominatorTree::Insert, BB, G.Dest});
+ Updates.push_back({DominatorTree::Delete, TableBB, G.Dest});
+ }
+
+ if (DTU)
+ DTU->applyUpdates(Updates);
+
+ // Everything below works on the block the switch now lives in.
+ BB = TableBB;
+ ++NumLookupTablesGuardedCase;
+ }
+
Builder.SetInsertPoint(SI);
// TableIndex is the switch condition - TableIndexOffset if we don't
// use the condition directly
diff --git a/llvm/test/Transforms/SimplifyCFG/X86/switch_to_lookup_table.ll b/llvm/test/Transforms/SimplifyCFG/X86/switch_to_lookup_table.ll
index cce11c12cb53e..c375bbe400849 100644
--- a/llvm/test/Transforms/SimplifyCFG/X86/switch_to_lookup_table.ll
+++ b/llvm/test/Transforms/SimplifyCFG/X86/switch_to_lookup_table.ll
@@ -39,6 +39,7 @@ target triple = "x86_64-unknown-linux-gnu"
; CHECK: @switch.table.covered_switch_with_bit_tests = private unnamed_addr constant [8 x i8] [i8 2, i8 2, i8 poison, i8 poison, i8 poison, i8 poison, i8 1, i8 1], align 4
; CHECK: @switch.table.signed_overflow1 = private unnamed_addr constant [4 x i16] [i16 3333, i16 4444, i16 1111, i16 2222], align 4
; CHECK: @switch.table.signed_overflow2 = private unnamed_addr constant [4 x i16] [i16 3333, i16 4444, i16 poison, i16 2222], align 4
+; CHECK: @switch.table.signed_overflow3 = private unnamed_addr constant [4 x i16] [i16 3333, i16 4444, i16 poison, i16 2222], align 4
; CHECK: @switch.table.constant_hole_unreachable_default_firstundef = private unnamed_addr constant [5 x i8] [i8 undef, i8 poison, i8 1, i8 1, i8 1], align 4
; CHECK: @switch.table.constant_hole_unreachable_default_lastundef = private unnamed_addr constant [5 x i8] [i8 1, i8 poison, i8 1, i8 1, i8 undef], align 4
; CHECK: @switch.table.linearmap_hole_unreachable_default = private unnamed_addr constant [5 x i8] [i8 1, i8 poison, i8 5, i8 7, i8 9], align 4
@@ -1807,24 +1808,18 @@ define i32 @signed_overflow3(i8 %n) {
; CHECK-LABEL: @signed_overflow3(
; CHECK-NEXT: start:
; CHECK-NEXT: [[TRUNC:%.*]] = trunc i8 [[N:%.*]] to i2
-; CHECK-NEXT: switch i2 [[TRUNC]], label [[START_UNREACHABLEDEFAULT:%.*]] [
-; CHECK-NEXT: i2 1, label [[BB6:%.*]]
-; CHECK-NEXT: i2 -2, label [[BB4:%.*]]
-; CHECK-NEXT: i2 -1, label [[BB5:%.*]]
-; CHECK-NEXT: i2 0, label [[BB1:%.*]]
-; CHECK-NEXT: ]
-; CHECK: start.unreachabledefault:
-; CHECK-NEXT: unreachable
+; CHECK-NEXT: [[COND:%.*]] = icmp eq i2 [[TRUNC]], 0
+; CHECK-NEXT: br i1 [[COND]], label [[BB1:%.*]], label [[START_LOOKUP:%.*]]
+; CHECK: start.lookup:
+; CHECK-NEXT: [[SWITCH_TABLEIDX:%.*]] = sub i2 [[TRUNC]], -2
+; CHECK-NEXT: [[TMP0:%.*]] = zext i2 [[SWITCH_TABLEIDX]] to i64
+; CHECK-NEXT: [[SWITCH_GEP:%.*]] = getelementptr inbounds [4 x i16], ptr @switch.table.signed_overflow3, i64 0, i64 [[TMP0]]
+; CHECK-NEXT: [[SWITCH_LOAD:%.*]] = load i16, ptr [[SWITCH_GEP]], align 2
+; CHECK-NEXT: [[SWITCH_EXT:%.*]] = zext i16 [[SWITCH_LOAD]] to i32
+; CHECK-NEXT: ret i32 [[SWITCH_EXT]]
; CHECK: bb1:
; CHECK-NEXT: call void @exit(i32 1)
; CHECK-NEXT: unreachable
-; CHECK: bb4:
-; CHECK-NEXT: br label [[BB6]]
-; CHECK: bb5:
-; CHECK-NEXT: br label [[BB6]]
-; CHECK: bb6:
-; CHECK-NEXT: [[DOTSROA_0_0:%.*]] = phi i32 [ 4444, [[BB5]] ], [ 3333, [[BB4]] ], [ 2222, [[START:%.*]] ]
-; CHECK-NEXT: ret i32 [[DOTSROA_0_0]]
;
start:
%trunc = trunc i8 %n to i2
diff --git a/llvm/test/Transforms/SimplifyCFG/switch-guard-non-table-case.ll b/llvm/test/Transforms/SimplifyCFG/switch-guard-non-table-case.ll
new file mode 100644
index 0000000000000..e9a667782bbe0
--- /dev/null
+++ b/llvm/test/Transforms/SimplifyCFG/switch-guard-non-table-case.ll
@@ -0,0 +1,961 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --check-globals all --version 6
+; One case that does not reach the destination the others do stops the whole
+; switch from becoming a lookup table. Guard that case out and build the table
+; from the rest.
+
+; RUN: opt < %s -passes='simplifycfg<switch-to-lookup>' -S \
+; RUN: -simplifycfg-require-and-preserve-domtree=1 \
+; RUN: | FileCheck %s --check-prefixes=CHECK
+; RUN: opt < %s -passes='simplifycfg<switch-to-lookup>' -S \
+; RUN: -simplifycfg-max-guarded-cases=0 \
+; RUN: -simplifycfg-require-and-preserve-domtree=1 \
+; RUN: | FileCheck %s --check-prefixes=OFF
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+declare void @sideeffect()
+declare i64 @use(i64)
+
+; The odd case is guarded and the rest become a table. No profile is needed: the
+; table is what the guard is for, and whether it is worth building is the
+; question shouldBuildLookupTable() already answers.
+
+;.
+; CHECK: @switch.table.guard_odd_case = private unnamed_addr constant [6 x i8] [i8 10, i8 poison, i8 12, i8 13, i8 14, i8 15], align 8
+; CHECK: @switch.table.guard_keeps_profile = private unnamed_addr constant [6 x i8] [i8 10, i8 poison, i8 12, i8 13, i8 14, i8 15], align 8
+; CHECK: @switch.table.guarded_slot_is_not_a_hole = private unnamed_addr constant [8 x i8] [i8 10, i8 11, i8 12, i8 poison, i8 14, i8 15, i8 16, i8 17], align 8
+; CHECK: @switch.table.guard_case_outside_range = private unnamed_addr constant [4 x i8] c"\0A\14!)", align 8
+; CHECK: @switch.table.unreachable_case_is_not_guarded = private unnamed_addr constant [6 x i8] c"\0A\00\0C\0D\0E\0F", align 8
+; CHECK: @switch.table.two_hot_odd_cases = private unnamed_addr constant [7 x i8] [i8 10, i8 poison, i8 12, i8 poison, i8 14, i8 15, i8 16], align 8
+;.
+; OFF: @switch.table.unreachable_case_is_not_guarded = private unnamed_addr constant [6 x i8] c"\0A\00\0C\0D\0E\0F", align 8
+;.
+define i64 @guard_odd_case(i8 %op) {
+; CHECK-LABEL: define i64 @guard_odd_case(
+; CHECK-SAME: i8 [[OP:%.*]]) {
+; CHECK-NEXT: [[ENTRY:.*:]]
+; CHECK-NEXT: [[COND:%.*]] = icmp eq i8 [[OP]], 1
+; CHECK-NEXT: br i1 [[COND]], label %[[ODD:.*]], label %[[ENTRY_LOOKUP:.*]]
+; CHECK: [[ENTRY_LOOKUP]]:
+; CHECK-NEXT: [[TMP0:%.*]] = icmp ult i8 [[OP]], 6
+; CHECK-NEXT: br i1 [[TMP0]], label %[[SWITCH_LOOKUP:.*]], label %[[JOIN:.*]]
+; CHECK: [[COMMON_RET:.*]]:
+; CHECK-NEXT: [[COMMON_RET_OP:%.*]] = phi i64 [ 999, %[[ODD]] ], [ [[R:%.*]], %[[JOIN]] ]
+; CHECK-NEXT: ret i64 [[COMMON_RET_OP]]
+; CHECK: [[ODD]]:
+; CHECK-NEXT: call void @sideeffect()
+; CHECK-NEXT: br label %[[COMMON_RET]]
+; CHECK: [[SWITCH_LOOKUP]]:
+; CHECK-NEXT: [[TMP1:%.*]] = zext nneg i8 [[OP]] to i64
+; CHECK-NEXT: [[SWITCH_GEP:%.*]] = getelementptr inbounds [6 x i8], ptr @switch.table.guard_odd_case, i64 0, i64 [[TMP1]]
+; CHECK-NEXT: [[SWITCH_LOAD:%.*]] = load i8, ptr [[SWITCH_GEP]], align 1
+; CHECK-NEXT: [[SWITCH_EXT:%.*]] = zext i8 [[SWITCH_LOAD]] to i64
+; CHECK-NEXT: br label %[[JOIN]]
+; CHECK: [[JOIN]]:
+; CHECK-NEXT: [[V:%.*]] = phi i64 [ [[SWITCH_EXT]], %[[SWITCH_LOOKUP]] ], [ 0, %[[ENTRY_LOOKUP]] ]
+; CHECK-NEXT: [[R]] = call i64 @use(i64 [[V]])
+; CHECK-NEXT: br label %[[COMMON_RET]]
+;
+; OFF-LABEL: define i64 @guard_odd_case(
+; OFF-SAME: i8 [[OP:%.*]]) {
+; OFF-NEXT: [[ENTRY:.*]]:
+; OFF-NEXT: switch i8 [[OP]], label %[[DEF:.*]] [
+; OFF-NEXT: i8 0, label %[[JOIN:.*]]
+; OFF-NEXT: i8 1, label %[[ODD:.*]]
+; OFF-NEXT: i8 2, label %[[C2:.*]]
+; OFF-NEXT: i8 3, label %[[C3:.*]]
+; OFF-NEXT: i8 4, label %[[C4:.*]]
+; OFF-NEXT: i8 5, label %[[C5:.*]]
+; OFF-NEXT: ]
+; OFF: [[C2]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[C3]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[C4]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[C5]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[DEF]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[COMMON_RET:.*]]:
+; OFF-NEXT: [[COMMON_RET_OP:%.*]] = phi i64 [ 999, %[[ODD]] ], [ [[R:%.*]], %[[JOIN]] ]
+; OFF-NEXT: ret i64 [[COMMON_RET_OP]]
+; OFF: [[ODD]]:
+; OFF-NEXT: call void @sideeffect()
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[JOIN]]:
+; OFF-NEXT: [[V:%.*]] = phi i64 [ 0, %[[DEF]] ], [ 12, %[[C2]] ], [ 13, %[[C3]] ], [ 14, %[[C4]] ], [ 15, %[[C5]] ], [ 10, %[[ENTRY]] ]
+; OFF-NEXT: [[R]] = call i64 @use(i64 [[V]])
+; OFF-NEXT: br label %[[COMMON_RET]]
+;
+entry:
+ switch i8 %op, label %def [
+ i8 0, label %c0
+ i8 1, label %odd
+ i8 2, label %c2
+ i8 3, label %c3
+ i8 4, label %c4
+ i8 5, label %c5
+ ]
+
+c0:
+ br label %join
+c2:
+ br label %join
+c3:
+ br label %join
+c4:
+ br label %join
+c5:
+ br label %join
+def:
+ br label %join
+
+odd:
+ call void @sideeffect()
+ ret i64 999
+
+join:
+ %v = phi i64 [ 10, %c0 ], [ 12, %c2 ], [ 13, %c3 ], [ 14, %c4 ], [ 15, %c5 ], [ 0, %def ]
+ %r = call i64 @use(i64 %v)
+ ret i64 %r
+}
+
+; The same when the odd case is the first one, which is the case that decides
+; which destination the others are measured against.
+
+define i64 @guard_first_case(i8 %op) {
+; CHECK-LABEL: define i64 @guard_first_case(
+; CHECK-SAME: i8 [[OP:%.*]]) {
+; CHECK-NEXT: [[ENTRY:.*:]]
+; CHECK-NEXT: [[COND:%.*]] = icmp eq i8 [[OP]], 0
+; CHECK-NEXT: br i1 [[COND]], label %[[ODD:.*]], label %[[ENTRY_LOOKUP:.*]]
+; CHECK: [[ENTRY_LOOKUP]]:
+; CHECK-NEXT: [[SWITCH_TABLEIDX:%.*]] = sub i8 [[OP]], 1
+; CHECK-NEXT: [[TMP0:%.*]] = icmp ult i8 [[SWITCH_TABLEIDX]], 5
+; CHECK-NEXT: [[SWITCH_IDX_CAST:%.*]] = zext i8 [[SWITCH_TABLEIDX]] to i64
+; CHECK-NEXT: [[SWITCH_OFFSET:%.*]] = add nsw i64 [[SWITCH_IDX_CAST]], 11
+; CHECK-NEXT: [[V:%.*]] = select i1 [[TMP0]], i64 [[SWITCH_OFFSET]], i64 0
+; CHECK-NEXT: [[R:%.*]] = call i64 @use(i64 [[V]])
+; CHECK-NEXT: br label %[[COMMON_RET:.*]]
+; CHECK: [[COMMON_RET]]:
+; CHECK-NEXT: [[COMMON_RET_OP:%.*]] = phi i64 [ 999, %[[ODD]] ], [ [[R]], %[[ENTRY_LOOKUP]] ]
+; CHECK-NEXT: ret i64 [[COMMON_RET_OP]]
+; CHECK: [[ODD]]:
+; CHECK-NEXT: call void @sideeffect()
+; CHECK-NEXT: br label %[[COMMON_RET]]
+;
+; OFF-LABEL: define i64 @guard_first_case(
+; OFF-SAME: i8 [[OP:%.*]]) {
+; OFF-NEXT: [[ENTRY:.*]]:
+; OFF-NEXT: switch i8 [[OP]], label %[[DEF:.*]] [
+; OFF-NEXT: i8 0, label %[[ODD:.*]]
+; OFF-NEXT: i8 1, label %[[JOIN:.*]]
+; OFF-NEXT: i8 2, label %[[C2:.*]]
+; OFF-NEXT: i8 3, label %[[C3:.*]]
+; OFF-NEXT: i8 4, label %[[C4:.*]]
+; OFF-NEXT: i8 5, label %[[C5:.*]]
+; OFF-NEXT: ]
+; OFF: [[C2]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[C3]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[C4]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[C5]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[DEF]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[COMMON_RET:.*]]:
+; OFF-NEXT: [[COMMON_RET_OP:%.*]] = phi i64 [ 999, %[[ODD]] ], [ [[R:%.*]], %[[JOIN]] ]
+; OFF-NEXT: ret i64 [[COMMON_RET_OP]]
+; OFF: [[ODD]]:
+; OFF-NEXT: call void @sideeffect()
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[JOIN]]:
+; OFF-NEXT: [[V:%.*]] = phi i64 [ 0, %[[DEF]] ], [ 12, %[[C2]] ], [ 13, %[[C3]] ], [ 14, %[[C4]] ], [ 15, %[[C5]] ], [ 11, %[[ENTRY]] ]
+; OFF-NEXT: [[R]] = call i64 @use(i64 [[V]])
+; OFF-NEXT: br label %[[COMMON_RET]]
+;
+entry:
+ switch i8 %op, label %def [
+ i8 0, label %odd
+ i8 1, label %c1
+ i8 2, label %c2
+ i8 3, label %c3
+ i8 4, label %c4
+ i8 5, label %c5
+ ]
+
+c1:
+ br label %join
+c2:
+ br label %join
+c3:
+ br label %join
+c4:
+ br label %join
+c5:
+ br label %join
+def:
+ br label %join
+
+odd:
+ call void @sideeffect()
+ ret i64 999
+
+join:
+ %v = phi i64 [ 11, %c1 ], [ 12, %c2 ], [ 13, %c3 ], [ 14, %c4 ], [ 15, %c5 ], [ 0, %def ]
+ %r = call i64 @use(i64 %v)
+ ret i64 %r
+}
+
+; Branch weights are carried across to both switches when they are there.
+
+define i64 @guard_keeps_profile(i8 %op) {
+; CHECK-LABEL: define i64 @guard_keeps_profile(
+; CHECK-SAME: i8 [[OP:%.*]]) {
+; CHECK-NEXT: [[ENTRY:.*:]]
+; CHECK-NEXT: [[COND:%.*]] = icmp eq i8 [[OP]], 1
+; CHECK-NEXT: br i1 [[COND]], label %[[ODD:.*]], label %[[ENTRY_LOOKUP:.*]], !prof [[PROF0:![0-9]+]]
+; CHECK: [[ENTRY_LOOKUP]]:
+; CHECK-NEXT: [[TMP0:%.*]] = icmp ult i8 [[OP]], 6
+; CHECK-NEXT: br i1 [[TMP0]], label %[[SWITCH_LOOKUP:.*]], label %[[JOIN:.*]], !prof [[PROF1:![0-9]+]]
+; CHECK: [[COMMON_RET:.*]]:
+; CHECK-NEXT: [[COMMON_RET_OP:%.*]] = phi i64 [ 999, %[[ODD]] ], [ [[R:%.*]], %[[JOIN]] ]
+; CHECK-NEXT: ret i64 [[COMMON_RET_OP]]
+; CHECK: [[ODD]]:
+; CHECK-NEXT: call void @sideeffect()
+; CHECK-NEXT: br label %[[COMMON_RET]]
+; CHECK: [[SWITCH_LOOKUP]]:
+; CHECK-NEXT: [[TMP1:%.*]] = zext nneg i8 [[OP]] to i64
+; CHECK-NEXT: [[SWITCH_GEP:%.*]] = getelementptr inbounds [6 x i8], ptr @switch.table.guard_keeps_profile, i64 0, i64 [[TMP1]]
+; CHECK-NEXT: [[SWITCH_LOAD:%.*]] = load i8, ptr [[SWITCH_GEP]], align 1
+; CHECK-NEXT: [[SWITCH_EXT:%.*]] = zext i8 [[SWITCH_LOAD]] to i64
+; CHECK-NEXT: br label %[[JOIN]]
+; CHECK: [[JOIN]]:
+; CHECK-NEXT: [[V:%.*]] = phi i64 [ [[SWITCH_EXT]], %[[SWITCH_LOOKUP]] ], [ 0, %[[ENTRY_LOOKUP]] ]
+; CHECK-NEXT: [[R]] = call i64 @use(i64 [[V]])
+; CHECK-NEXT: br label %[[COMMON_RET]]
+;
+; OFF-LABEL: define i64 @guard_keeps_profile(
+; OFF-SAME: i8 [[OP:%.*]]) {
+; OFF-NEXT: [[ENTRY:.*]]:
+; OFF-NEXT: switch i8 [[OP]], label %[[DEF:.*]] [
+; OFF-NEXT: i8 0, label %[[JOIN:.*]]
+; OFF-NEXT: i8 1, label %[[ODD:.*]]
+; OFF-NEXT: i8 2, label %[[C2:.*]]
+; OFF-NEXT: i8 3, label %[[C3:.*]]
+; OFF-NEXT: i8 4, label %[[C4:.*]]
+; OFF-NEXT: i8 5, label %[[C5:.*]]
+; OFF-NEXT: ], !prof [[PROF0:![0-9]+]]
+; OFF: [[C2]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[C3]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[C4]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[C5]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[DEF]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[COMMON_RET:.*]]:
+; OFF-NEXT: [[COMMON_RET_OP:%.*]] = phi i64 [ 999, %[[ODD]] ], [ [[R:%.*]], %[[JOIN]] ]
+; OFF-NEXT: ret i64 [[COMMON_RET_OP]]
+; OFF: [[ODD]]:
+; OFF-NEXT: call void @sideeffect()
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[JOIN]]:
+; OFF-NEXT: [[V:%.*]] = phi i64 [ 0, %[[DEF]] ], [ 12, %[[C2]] ], [ 13, %[[C3]] ], [ 14, %[[C4]] ], [ 15, %[[C5]] ], [ 10, %[[ENTRY]] ]
+; OFF-NEXT: [[R]] = call i64 @use(i64 [[V]])
+; OFF-NEXT: br label %[[COMMON_RET]]
+;
+entry:
+ switch i8 %op, label %def [
+ i8 0, label %c0
+ i8 1, label %odd
+ i8 2, label %c2
+ i8 3, label %c3
+ i8 4, label %c4
+ i8 5, label %c5
+ ], !prof !0
+
+c0:
+ br label %join
+c2:
+ br label %join
+c3:
+ br label %join
+c4:
+ br label %join
+c5:
+ br label %join
+def:
+ br label %join
+
+odd:
+ call void @sideeffect()
+ ret i64 999
+
+join:
+ %v = phi i64 [ 10, %c0 ], [ 12, %c2 ], [ 13, %c3 ], [ 14, %c4 ], [ 15, %c5 ], [ 0, %def ]
+ %r = call i64 @use(i64 %v)
+ ret i64 %r
+}
+
+; The slot the guarded case leaves behind is never loaded, but leaving it empty
+; would make the cases that do reach the table pay for a hole check. It takes a
+; value the table already holds instead, and no mask is needed.
+
+define i64 @guarded_slot_is_not_a_hole(i8 %op, i64 %d) {
+; CHECK-LABEL: define i64 @guarded_slot_is_not_a_hole(
+; CHECK-SAME: i8 [[OP:%.*]], i64 [[D:%.*]]) {
+; CHECK-NEXT: [[ENTRY:.*:]]
+; CHECK-NEXT: [[COND:%.*]] = icmp eq i8 [[OP]], 3
+; CHECK-NEXT: br i1 [[COND]], label %[[ODD:.*]], label %[[ENTRY_LOOKUP:.*]]
+; CHECK: [[ENTRY_LOOKUP]]:
+; CHECK-NEXT: [[TMP0:%.*]] = icmp ult i8 [[OP]], 8
+; CHECK-NEXT: br i1 [[TMP0]], label %[[SWITCH_LOOKUP:.*]], label %[[JOIN:.*]]
+; CHECK: [[COMMON_RET:.*]]:
+; CHECK-NEXT: [[COMMON_RET_OP:%.*]] = phi i64 [ 999, %[[ODD]] ], [ [[R:%.*]], %[[JOIN]] ]
+; CHECK-NEXT: ret i64 [[COMMON_RET_OP]]
+; CHECK: [[ODD]]:
+; CHECK-NEXT: call void @sideeffect()
+; CHECK-NEXT: br label %[[COMMON_RET]]
+; CHECK: [[SWITCH_LOOKUP]]:
+; CHECK-NEXT: [[TMP1:%.*]] = zext nneg i8 [[OP]] to i64
+; CHECK-NEXT: [[SWITCH_GEP:%.*]] = getelementptr inbounds [8 x i8], ptr @switch.table.guarded_slot_is_not_a_hole, i64 0, i64 [[TMP1]]
+; CHECK-NEXT: [[SWITCH_LOAD:%.*]] = load i8, ptr [[SWITCH_GEP]], align 1
+; CHECK-NEXT: [[SWITCH_EXT:%.*]] = zext i8 [[SWITCH_LOAD]] to i64
+; CHECK-NEXT: br label %[[JOIN]]
+; CHECK: [[JOIN]]:
+; CHECK-NEXT: [[V:%.*]] = phi i64 [ [[SWITCH_EXT]], %[[SWITCH_LOOKUP]] ], [ [[D]], %[[ENTRY_LOOKUP]] ]
+; CHECK-NEXT: [[R]] = call i64 @use(i64 [[V]])
+; CHECK-NEXT: br label %[[COMMON_RET]]
+;
+; OFF-LABEL: define i64 @guarded_slot_is_not_a_hole(
+; OFF-SAME: i8 [[OP:%.*]], i64 [[D:%.*]]) {
+; OFF-NEXT: [[ENTRY:.*]]:
+; OFF-NEXT: switch i8 [[OP]], label %[[DEF:.*]] [
+; OFF-NEXT: i8 0, label %[[JOIN:.*]]
+; OFF-NEXT: i8 1, label %[[C1:.*]]
+; OFF-NEXT: i8 2, label %[[C2:.*]]
+; OFF-NEXT: i8 3, label %[[ODD:.*]]
+; OFF-NEXT: i8 4, label %[[C4:.*]]
+; OFF-NEXT: i8 5, label %[[C5:.*]]
+; OFF-NEXT: i8 6, label %[[C6:.*]]
+; OFF-NEXT: i8 7, label %[[C7:.*]]
+; OFF-NEXT: ]
+; OFF: [[C1]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[C2]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[C4]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[C5]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[C6]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[C7]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[DEF]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[COMMON_RET:.*]]:
+; OFF-NEXT: [[COMMON_RET_OP:%.*]] = phi i64 [ 999, %[[ODD]] ], [ [[R:%.*]], %[[JOIN]] ]
+; OFF-NEXT: ret i64 [[COMMON_RET_OP]]
+; OFF: [[ODD]]:
+; OFF-NEXT: call void @sideeffect()
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[JOIN]]:
+; OFF-NEXT: [[V:%.*]] = phi i64 [ [[D]], %[[DEF]] ], [ 11, %[[C1]] ], [ 12, %[[C2]] ], [ 14, %[[C4]] ], [ 15, %[[C5]] ], [ 16, %[[C6]] ], [ 17, %[[C7]] ], [ 10, %[[ENTRY]] ]
+; OFF-NEXT: [[R]] = call i64 @use(i64 [[V]])
+; OFF-NEXT: br label %[[COMMON_RET]]
+;
+entry:
+ switch i8 %op, label %def [
+ i8 0, label %c0
+ i8 1, label %c1
+ i8 2, label %c2
+ i8 3, label %odd
+ i8 4, label %c4
+ i8 5, label %c5
+ i8 6, label %c6
+ i8 7, label %c7
+ ]
+
+c0:
+ br label %join
+c1:
+ br label %join
+c2:
+ br label %join
+c4:
+ br label %join
+c5:
+ br label %join
+c6:
+ br label %join
+c7:
+ br label %join
+def:
+ br label %join
+
+odd:
+ call void @sideeffect()
+ ret i64 999
+
+join:
+ %v = phi i64 [ 10, %c0 ], [ 11, %c1 ], [ 12, %c2 ], [ 14, %c4 ], [ 15, %c5 ], [ 16, %c6 ], [ 17, %c7 ], [ %d, %def ]
+ %r = call i64 @use(i64 %v)
+ ret i64 %r
+}
+
+; A guarded case outside the range the others span owns no slot at all, and must
+; not stretch the table to cover one.
+
+define i64 @guard_case_outside_range(i8 %op) {
+; CHECK-LABEL: define i64 @guard_case_outside_range(
+; CHECK-SAME: i8 [[OP:%.*]]) {
+; CHECK-NEXT: [[ENTRY:.*:]]
+; CHECK-NEXT: [[COND:%.*]] = icmp eq i8 [[OP]], 100
+; CHECK-NEXT: br i1 [[COND]], label %[[ODD:.*]], label %[[ENTRY_LOOKUP:.*]]
+; CHECK: [[ENTRY_LOOKUP]]:
+; CHECK-NEXT: [[TMP0:%.*]] = icmp ult i8 [[OP]], 4
+; CHECK-NEXT: br i1 [[TMP0]], label %[[SWITCH_LOOKUP:.*]], label %[[JOIN:.*]]
+; CHECK: [[COMMON_RET:.*]]:
+; CHECK-NEXT: [[COMMON_RET_OP:%.*]] = phi i64 [ 999, %[[ODD]] ], [ [[R:%.*]], %[[JOIN]] ]
+; CHECK-NEXT: ret i64 [[COMMON_RET_OP]]
+; CHECK: [[ODD]]:
+; CHECK-NEXT: call void @sideeffect()
+; CHECK-NEXT: br label %[[COMMON_RET]]
+; CHECK: [[SWITCH_LOOKUP]]:
+; CHECK-NEXT: [[TMP1:%.*]] = zext nneg i8 [[OP]] to i64
+; CHECK-NEXT: [[SWITCH_GEP:%.*]] = getelementptr inbounds [4 x i8], ptr @switch.table.guard_case_outside_range, i64 0, i64 [[TMP1]]
+; CHECK-NEXT: [[SWITCH_LOAD:%.*]] = load i8, ptr [[SWITCH_GEP]], align 1
+; CHECK-NEXT: [[SWITCH_EXT:%.*]] = zext i8 [[SWITCH_LOAD]] to i64
+; CHECK-NEXT: br label %[[JOIN]]
+; CHECK: [[JOIN]]:
+; CHECK-NEXT: [[V:%.*]] = phi i64 [ [[SWITCH_EXT]], %[[SWITCH_LOOKUP]] ], [ 0, %[[ENTRY_LOOKUP]] ]
+; CHECK-NEXT: [[R]] = call i64 @use(i64 [[V]])
+; CHECK-NEXT: br label %[[COMMON_RET]]
+;
+; OFF-LABEL: define i64 @guard_case_outside_range(
+; OFF-SAME: i8 [[OP:%.*]]) {
+; OFF-NEXT: [[ENTRY:.*]]:
+; OFF-NEXT: switch i8 [[OP]], label %[[DEF:.*]] [
+; OFF-NEXT: i8 0, label %[[JOIN:.*]]
+; OFF-NEXT: i8 1, label %[[C1:.*]]
+; OFF-NEXT: i8 2, label %[[C2:.*]]
+; OFF-NEXT: i8 3, label %[[C3:.*]]
+; OFF-NEXT: i8 100, label %[[ODD:.*]]
+; OFF-NEXT: ]
+; OFF: [[C1]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[C2]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[C3]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[DEF]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[COMMON_RET:.*]]:
+; OFF-NEXT: [[COMMON_RET_OP:%.*]] = phi i64 [ 999, %[[ODD]] ], [ [[R:%.*]], %[[JOIN]] ]
+; OFF-NEXT: ret i64 [[COMMON_RET_OP]]
+; OFF: [[ODD]]:
+; OFF-NEXT: call void @sideeffect()
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[JOIN]]:
+; OFF-NEXT: [[V:%.*]] = phi i64 [ 0, %[[DEF]] ], [ 20, %[[C1]] ], [ 33, %[[C2]] ], [ 41, %[[C3]] ], [ 10, %[[ENTRY]] ]
+; OFF-NEXT: [[R]] = call i64 @use(i64 [[V]])
+; OFF-NEXT: br label %[[COMMON_RET]]
+;
+entry:
+ switch i8 %op, label %def [
+ i8 0, label %c0
+ i8 1, label %c1
+ i8 2, label %c2
+ i8 3, label %c3
+ i8 100, label %odd
+ ]
+
+c0:
+ br label %join
+c1:
+ br label %join
+c2:
+ br label %join
+c3:
+ br label %join
+def:
+ br label %join
+
+odd:
+ call void @sideeffect()
+ ret i64 999
+
+join:
+ %v = phi i64 [ 10, %c0 ], [ 20, %c1 ], [ 33, %c2 ], [ 41, %c3 ], [ 0, %def ]
+ %r = call i64 @use(i64 %v)
+ ret i64 %r
+}
+
+; Only one case can be guarded: a second one costs the table.
+
+define i64 @two_odd_cases(i8 %op) {
+; CHECK-LABEL: define i64 @two_odd_cases(
+; CHECK-SAME: i8 [[OP:%.*]]) {
+; CHECK-NEXT: [[ENTRY:.*]]:
+; CHECK-NEXT: switch i8 [[OP]], label %[[DEF:.*]] [
+; CHECK-NEXT: i8 0, label %[[JOIN:.*]]
+; CHECK-NEXT: i8 1, label %[[ODD:.*]]
+; CHECK-NEXT: i8 2, label %[[ODD2:.*]]
+; CHECK-NEXT: i8 3, label %[[C3:.*]]
+; CHECK-NEXT: i8 4, label %[[C4:.*]]
+; CHECK-NEXT: i8 5, label %[[C5:.*]]
+; CHECK-NEXT: ]
+; CHECK: [[C3]]:
+; CHECK-NEXT: br label %[[JOIN]]
+; CHECK: [[C4]]:
+; CHECK-NEXT: br label %[[JOIN]]
+; CHECK: [[C5]]:
+; CHECK-NEXT: br label %[[JOIN]]
+; CHECK: [[DEF]]:
+; CHECK-NEXT: br label %[[JOIN]]
+; CHECK: [[COMMON_RET:.*]]:
+; CHECK-NEXT: [[COMMON_RET_OP:%.*]] = phi i64 [ 999, %[[ODD]] ], [ 998, %[[ODD2]] ], [ [[R:%.*]], %[[JOIN]] ]
+; CHECK-NEXT: ret i64 [[COMMON_RET_OP]]
+; CHECK: [[ODD]]:
+; CHECK-NEXT: call void @sideeffect()
+; CHECK-NEXT: br label %[[COMMON_RET]]
+; CHECK: [[ODD2]]:
+; CHECK-NEXT: call void @sideeffect()
+; CHECK-NEXT: br label %[[COMMON_RET]]
+; CHECK: [[JOIN]]:
+; CHECK-NEXT: [[V:%.*]] = phi i64 [ 0, %[[DEF]] ], [ 13, %[[C3]] ], [ 14, %[[C4]] ], [ 15, %[[C5]] ], [ 10, %[[ENTRY]] ]
+; CHECK-NEXT: [[R]] = call i64 @use(i64 [[V]])
+; CHECK-NEXT: br label %[[COMMON_RET]]
+;
+; OFF-LABEL: define i64 @two_odd_cases(
+; OFF-SAME: i8 [[OP:%.*]]) {
+; OFF-NEXT: [[ENTRY:.*]]:
+; OFF-NEXT: switch i8 [[OP]], label %[[DEF:.*]] [
+; OFF-NEXT: i8 0, label %[[JOIN:.*]]
+; OFF-NEXT: i8 1, label %[[ODD:.*]]
+; OFF-NEXT: i8 2, label %[[ODD2:.*]]
+; OFF-NEXT: i8 3, label %[[C3:.*]]
+; OFF-NEXT: i8 4, label %[[C4:.*]]
+; OFF-NEXT: i8 5, label %[[C5:.*]]
+; OFF-NEXT: ]
+; OFF: [[C3]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[C4]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[C5]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[DEF]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[COMMON_RET:.*]]:
+; OFF-NEXT: [[COMMON_RET_OP:%.*]] = phi i64 [ 999, %[[ODD]] ], [ 998, %[[ODD2]] ], [ [[R:%.*]], %[[JOIN]] ]
+; OFF-NEXT: ret i64 [[COMMON_RET_OP]]
+; OFF: [[ODD]]:
+; OFF-NEXT: call void @sideeffect()
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[ODD2]]:
+; OFF-NEXT: call void @sideeffect()
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[JOIN]]:
+; OFF-NEXT: [[V:%.*]] = phi i64 [ 0, %[[DEF]] ], [ 13, %[[C3]] ], [ 14, %[[C4]] ], [ 15, %[[C5]] ], [ 10, %[[ENTRY]] ]
+; OFF-NEXT: [[R]] = call i64 @use(i64 [[V]])
+; OFF-NEXT: br label %[[COMMON_RET]]
+;
+entry:
+ switch i8 %op, label %def [
+ i8 0, label %c0
+ i8 1, label %odd
+ i8 2, label %odd2
+ i8 3, label %c3
+ i8 4, label %c4
+ i8 5, label %c5
+ ]
+
+c0:
+ br label %join
+c3:
+ br label %join
+c4:
+ br label %join
+c5:
+ br label %join
+def:
+ br label %join
+
+odd:
+ call void @sideeffect()
+ ret i64 999
+odd2:
+ call void @sideeffect()
+ ret i64 998
+
+join:
+ %v = phi i64 [ 10, %c0 ], [ 13, %c3 ], [ 14, %c4 ], [ 15, %c5 ], [ 0, %def ]
+ %r = call i64 @use(i64 %v)
+ ret i64 %r
+}
+
+; Too few cases would be left for a table to be worth building.
+
+define i64 @too_few_cases_left(i8 %op) {
+; CHECK-LABEL: define i64 @too_few_cases_left(
+; CHECK-SAME: i8 [[OP:%.*]]) {
+; CHECK-NEXT: [[ENTRY:.*]]:
+; CHECK-NEXT: switch i8 [[OP]], label %[[DEF:.*]] [
+; CHECK-NEXT: i8 0, label %[[JOIN:.*]]
+; CHECK-NEXT: i8 1, label %[[ODD:.*]]
+; CHECK-NEXT: i8 2, label %[[C2:.*]]
+; CHECK-NEXT: ]
+; CHECK: [[C2]]:
+; CHECK-NEXT: br label %[[JOIN]]
+; CHECK: [[DEF]]:
+; CHECK-NEXT: br label %[[JOIN]]
+; CHECK: [[COMMON_RET:.*]]:
+; CHECK-NEXT: [[COMMON_RET_OP:%.*]] = phi i64 [ 999, %[[ODD]] ], [ [[R:%.*]], %[[JOIN]] ]
+; CHECK-NEXT: ret i64 [[COMMON_RET_OP]]
+; CHECK: [[ODD]]:
+; CHECK-NEXT: call void @sideeffect()
+; CHECK-NEXT: br label %[[COMMON_RET]]
+; CHECK: [[JOIN]]:
+; CHECK-NEXT: [[V:%.*]] = phi i64 [ 0, %[[DEF]] ], [ 12, %[[C2]] ], [ 10, %[[ENTRY]] ]
+; CHECK-NEXT: [[R]] = call i64 @use(i64 [[V]])
+; CHECK-NEXT: br label %[[COMMON_RET]]
+;
+; OFF-LABEL: define i64 @too_few_cases_left(
+; OFF-SAME: i8 [[OP:%.*]]) {
+; OFF-NEXT: [[ENTRY:.*]]:
+; OFF-NEXT: switch i8 [[OP]], label %[[DEF:.*]] [
+; OFF-NEXT: i8 0, label %[[JOIN:.*]]
+; OFF-NEXT: i8 1, label %[[ODD:.*]]
+; OFF-NEXT: i8 2, label %[[C2:.*]]
+; OFF-NEXT: ]
+; OFF: [[C2]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[DEF]]:
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[COMMON_RET:.*]]:
+; OFF-NEXT: [[COMMON_RET_OP:%.*]] = phi i64 [ 999, %[[ODD]] ], [ [[R:%.*]], %[[JOIN]] ]
+; OFF-NEXT: ret i64 [[COMMON_RET_OP]]
+; OFF: [[ODD]]:
+; OFF-NEXT: call void @sideeffect()
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[JOIN]]:
+; OFF-NEXT: [[V:%.*]] = phi i64 [ 0, %[[DEF]] ], [ 12, %[[C2]] ], [ 10, %[[ENTRY]] ]
+; OFF-NEXT: [[R]] = call i64 @use(i64 [[V]])
+; OFF-NEXT: br label %[[COMMON_RET]]
+;
+entry:
+ switch i8 %op, label %def [
+ i8 0, label %c0
+ i8 1, label %odd
+ i8 2, label %c2
+ ]
+
+c0:
+ br label %join
+c2:
+ br label %join
+def:
+ br label %join
+
+odd:
+ call void @sideeffect()
+ ret i64 999
+
+join:
+ %v = phi i64 [ 10, %c0 ], [ 12, %c2 ], [ 0, %def ]
+ %r = call i64 @use(i64 %v)
+ ret i64 %r
+}
+
+; A case that cannot be reached is not worth a test of its own; SimplifyCFG
+; removes it in its own time and the table fills the hole it leaves.
+
+define i64 @unreachable_case_is_not_guarded(i8 %op) {
+; CHECK-LABEL: define i64 @unreachable_case_is_not_guarded(
+; CHECK-SAME: i8 [[OP:%.*]]) {
+; CHECK-NEXT: [[ENTRY:.*]]:
+; CHECK-NEXT: [[TMP0:%.*]] = icmp ult i8 [[OP]], 6
+; CHECK-NEXT: br i1 [[TMP0]], label %[[SWITCH_LOOKUP:.*]], label %[[JOIN:.*]]
+; CHECK: [[SWITCH_LOOKUP]]:
+; CHECK-NEXT: [[TMP1:%.*]] = zext nneg i8 [[OP]] to i64
+; CHECK-NEXT: [[SWITCH_GEP:%.*]] = getelementptr inbounds [6 x i8], ptr @switch.table.unreachable_case_is_not_guarded, i64 0, i64 [[TMP1]]
+; CHECK-NEXT: [[SWITCH_LOAD:%.*]] = load i8, ptr [[SWITCH_GEP]], align 1
+; CHECK-NEXT: [[SWITCH_EXT:%.*]] = zext i8 [[SWITCH_LOAD]] to i64
+; CHECK-NEXT: br label %[[JOIN]]
+; CHECK: [[JOIN]]:
+; CHECK-NEXT: [[V:%.*]] = phi i64 [ [[SWITCH_EXT]], %[[SWITCH_LOOKUP]] ], [ 0, %[[ENTRY]] ]
+; CHECK-NEXT: [[R:%.*]] = call i64 @use(i64 [[V]])
+; CHECK-NEXT: ret i64 [[R]]
+;
+; OFF-LABEL: define i64 @unreachable_case_is_not_guarded(
+; OFF-SAME: i8 [[OP:%.*]]) {
+; OFF-NEXT: [[ENTRY:.*]]:
+; OFF-NEXT: [[TMP0:%.*]] = icmp ult i8 [[OP]], 6
+; OFF-NEXT: br i1 [[TMP0]], label %[[SWITCH_LOOKUP:.*]], label %[[JOIN:.*]]
+; OFF: [[SWITCH_LOOKUP]]:
+; OFF-NEXT: [[TMP1:%.*]] = zext nneg i8 [[OP]] to i64
+; OFF-NEXT: [[SWITCH_GEP:%.*]] = getelementptr inbounds [6 x i8], ptr @switch.table.unreachable_case_is_not_guarded, i64 0, i64 [[TMP1]]
+; OFF-NEXT: [[SWITCH_LOAD:%.*]] = load i8, ptr [[SWITCH_GEP]], align 1
+; OFF-NEXT: [[SWITCH_EXT:%.*]] = zext i8 [[SWITCH_LOAD]] to i64
+; OFF-NEXT: br label %[[JOIN]]
+; OFF: [[JOIN]]:
+; OFF-NEXT: [[V:%.*]] = phi i64 [ [[SWITCH_EXT]], %[[SWITCH_LOOKUP]] ], [ 0, %[[ENTRY]] ]
+; OFF-NEXT: [[R:%.*]] = call i64 @use(i64 [[V]])
+; OFF-NEXT: ret i64 [[R]]
+;
+entry:
+ switch i8 %op, label %def [
+ i8 0, label %c0
+ i8 1, label %dead
+ i8 2, label %c2
+ i8 3, label %c3
+ i8 4, label %c4
+ i8 5, label %c5
+ ]
+
+c0:
+ br label %join
+c2:
+ br label %join
+c3:
+ br label %join
+c4:
+ br label %join
+c5:
+ br label %join
+def:
+ br label %join
+
+dead:
+ unreachable
+
+join:
+ %v = phi i64 [ 10, %c0 ], [ 12, %c2 ], [ 13, %c3 ], [ 14, %c4 ], [ 15, %c5 ], [ 0, %def ]
+ %r = call i64 @use(i64 %v)
+ ret i64 %r
+}
+
+!0 = !{!"branch_weights", i32 1, i32 10, i32 1000, i32 10, i32 10, i32 10, i32 10}
+
+; Two cases can be guarded when the profile says both are taken more often than
+; the average case that reaches the table.
+define i64 @two_hot_odd_cases(i8 %op) {
+; CHECK-LABEL: define i64 @two_hot_odd_cases(
+; CHECK-SAME: i8 [[OP:%.*]]) {
+; CHECK-NEXT: [[ENTRY:.*:]]
+; CHECK-NEXT: switch i8 [[OP]], label %[[ENTRY_LOOKUP:.*]] [
+; CHECK-NEXT: i8 1, label %[[HOT1:.*]]
+; CHECK-NEXT: i8 3, label %[[HOT3:.*]]
+; CHECK-NEXT: ], !prof [[PROF2:![0-9]+]]
+; CHECK: [[ENTRY_LOOKUP]]:
+; CHECK-NEXT: [[TMP0:%.*]] = icmp ult i8 [[OP]], 7
+; CHECK-NEXT: br i1 [[TMP0]], label %[[SWITCH_LOOKUP:.*]], label %[[COMMON_RET:.*]], !prof [[PROF1]]
+; CHECK: [[COMMON_RET]]:
+; CHECK-NEXT: [[COMMON_RET_OP:%.*]] = phi i64 [ 1, %[[HOT1]] ], [ 3, %[[HOT3]] ], [ [[SWITCH_EXT:%.*]], %[[SWITCH_LOOKUP]] ], [ 99, %[[ENTRY_LOOKUP]] ]
+; CHECK-NEXT: ret i64 [[COMMON_RET_OP]]
+; CHECK: [[HOT1]]:
+; CHECK-NEXT: call void @sideeffect()
+; CHECK-NEXT: br label %[[COMMON_RET]]
+; CHECK: [[HOT3]]:
+; CHECK-NEXT: call void @sideeffect()
+; CHECK-NEXT: br label %[[COMMON_RET]]
+; CHECK: [[SWITCH_LOOKUP]]:
+; CHECK-NEXT: [[TMP1:%.*]] = zext nneg i8 [[OP]] to i64
+; CHECK-NEXT: [[SWITCH_GEP:%.*]] = getelementptr inbounds [7 x i8], ptr @switch.table.two_hot_odd_cases, i64 0, i64 [[TMP1]]
+; CHECK-NEXT: [[SWITCH_LOAD:%.*]] = load i8, ptr [[SWITCH_GEP]], align 1
+; CHECK-NEXT: [[SWITCH_EXT]] = zext i8 [[SWITCH_LOAD]] to i64
+; CHECK-NEXT: br label %[[COMMON_RET]]
+;
+; OFF-LABEL: define i64 @two_hot_odd_cases(
+; OFF-SAME: i8 [[OP:%.*]]) {
+; OFF-NEXT: [[ENTRY:.*]]:
+; OFF-NEXT: switch i8 [[OP]], label %[[DEF:.*]] [
+; OFF-NEXT: i8 0, label %[[COMMON_RET:.*]]
+; OFF-NEXT: i8 1, label %[[HOT1:.*]]
+; OFF-NEXT: i8 2, label %[[C2:.*]]
+; OFF-NEXT: i8 3, label %[[HOT3:.*]]
+; OFF-NEXT: i8 4, label %[[C4:.*]]
+; OFF-NEXT: i8 5, label %[[C5:.*]]
+; OFF-NEXT: i8 6, label %[[C6:.*]]
+; OFF-NEXT: ], !prof [[PROF1:![0-9]+]]
+; OFF: [[C2]]:
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[C4]]:
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[C5]]:
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[C6]]:
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[DEF]]:
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[COMMON_RET]]:
+; OFF-NEXT: [[COMMON_RET_OP:%.*]] = phi i64 [ 1, %[[HOT1]] ], [ 3, %[[HOT3]] ], [ 99, %[[DEF]] ], [ 12, %[[C2]] ], [ 14, %[[C4]] ], [ 15, %[[C5]] ], [ 16, %[[C6]] ], [ 10, %[[ENTRY]] ]
+; OFF-NEXT: ret i64 [[COMMON_RET_OP]]
+; OFF: [[HOT1]]:
+; OFF-NEXT: call void @sideeffect()
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[HOT3]]:
+; OFF-NEXT: call void @sideeffect()
+; OFF-NEXT: br label %[[COMMON_RET]]
+;
+entry:
+ switch i8 %op, label %def [
+ i8 0, label %c0
+ i8 1, label %hot1
+ i8 2, label %c2
+ i8 3, label %hot3
+ i8 4, label %c4
+ i8 5, label %c5
+ i8 6, label %c6
+ ], !prof !1
+
+c0:
+ br label %join
+c2:
+ br label %join
+c4:
+ br label %join
+c5:
+ br label %join
+c6:
+ br label %join
+def:
+ br label %join
+
+hot1:
+ call void @sideeffect()
+ ret i64 1
+
+hot3:
+ call void @sideeffect()
+ ret i64 3
+
+join:
+ %v = phi i64 [ 10, %c0 ], [ 12, %c2 ], [ 14, %c4 ], [ 15, %c5 ], [ 16, %c6 ], [ 99, %def ]
+ ret i64 %v
+}
+
+; The same switch, but the odd cases are colder than the average case reaching
+; the table: they do not earn a test of their own and the transform gives up.
+define i64 @two_cold_odd_cases(i8 %op) {
+; CHECK-LABEL: define i64 @two_cold_odd_cases(
+; CHECK-SAME: i8 [[OP:%.*]]) {
+; CHECK-NEXT: [[ENTRY:.*]]:
+; CHECK-NEXT: switch i8 [[OP]], label %[[DEF:.*]] [
+; CHECK-NEXT: i8 0, label %[[COMMON_RET:.*]]
+; CHECK-NEXT: i8 1, label %[[COLD1:.*]]
+; CHECK-NEXT: i8 2, label %[[C2:.*]]
+; CHECK-NEXT: i8 3, label %[[COLD3:.*]]
+; CHECK-NEXT: i8 4, label %[[C4:.*]]
+; CHECK-NEXT: i8 5, label %[[C5:.*]]
+; CHECK-NEXT: i8 6, label %[[C6:.*]]
+; CHECK-NEXT: ], !prof [[PROF3:![0-9]+]]
+; CHECK: [[C2]]:
+; CHECK-NEXT: br label %[[COMMON_RET]]
+; CHECK: [[C4]]:
+; CHECK-NEXT: br label %[[COMMON_RET]]
+; CHECK: [[C5]]:
+; CHECK-NEXT: br label %[[COMMON_RET]]
+; CHECK: [[C6]]:
+; CHECK-NEXT: br label %[[COMMON_RET]]
+; CHECK: [[DEF]]:
+; CHECK-NEXT: br label %[[COMMON_RET]]
+; CHECK: [[COMMON_RET]]:
+; CHECK-NEXT: [[COMMON_RET_OP:%.*]] = phi i64 [ 1, %[[COLD1]] ], [ 3, %[[COLD3]] ], [ 99, %[[DEF]] ], [ 12, %[[C2]] ], [ 14, %[[C4]] ], [ 15, %[[C5]] ], [ 16, %[[C6]] ], [ 10, %[[ENTRY]] ]
+; CHECK-NEXT: ret i64 [[COMMON_RET_OP]]
+; CHECK: [[COLD1]]:
+; CHECK-NEXT: call void @sideeffect()
+; CHECK-NEXT: br label %[[COMMON_RET]]
+; CHECK: [[COLD3]]:
+; CHECK-NEXT: call void @sideeffect()
+; CHECK-NEXT: br label %[[COMMON_RET]]
+;
+; OFF-LABEL: define i64 @two_cold_odd_cases(
+; OFF-SAME: i8 [[OP:%.*]]) {
+; OFF-NEXT: [[ENTRY:.*]]:
+; OFF-NEXT: switch i8 [[OP]], label %[[DEF:.*]] [
+; OFF-NEXT: i8 0, label %[[COMMON_RET:.*]]
+; OFF-NEXT: i8 1, label %[[COLD1:.*]]
+; OFF-NEXT: i8 2, label %[[C2:.*]]
+; OFF-NEXT: i8 3, label %[[COLD3:.*]]
+; OFF-NEXT: i8 4, label %[[C4:.*]]
+; OFF-NEXT: i8 5, label %[[C5:.*]]
+; OFF-NEXT: i8 6, label %[[C6:.*]]
+; OFF-NEXT: ], !prof [[PROF2:![0-9]+]]
+; OFF: [[C2]]:
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[C4]]:
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[C5]]:
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[C6]]:
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[DEF]]:
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[COMMON_RET]]:
+; OFF-NEXT: [[COMMON_RET_OP:%.*]] = phi i64 [ 1, %[[COLD1]] ], [ 3, %[[COLD3]] ], [ 99, %[[DEF]] ], [ 12, %[[C2]] ], [ 14, %[[C4]] ], [ 15, %[[C5]] ], [ 16, %[[C6]] ], [ 10, %[[ENTRY]] ]
+; OFF-NEXT: ret i64 [[COMMON_RET_OP]]
+; OFF: [[COLD1]]:
+; OFF-NEXT: call void @sideeffect()
+; OFF-NEXT: br label %[[COMMON_RET]]
+; OFF: [[COLD3]]:
+; OFF-NEXT: call void @sideeffect()
+; OFF-NEXT: br label %[[COMMON_RET]]
+;
+entry:
+ switch i8 %op, label %def [
+ i8 0, label %c0
+ i8 1, label %cold1
+ i8 2, label %c2
+ i8 3, label %cold3
+ i8 4, label %c4
+ i8 5, label %c5
+ i8 6, label %c6
+ ], !prof !2
+
+c0:
+ br label %join
+c2:
+ br label %join
+c4:
+ br label %join
+c5:
+ br label %join
+c6:
+ br label %join
+def:
+ br label %join
+
+cold1:
+ call void @sideeffect()
+ ret i64 1
+
+cold3:
+ call void @sideeffect()
+ ret i64 3
+
+join:
+ %v = phi i64 [ 10, %c0 ], [ 12, %c2 ], [ 14, %c4 ], [ 15, %c5 ], [ 16, %c6 ], [ 99, %def ]
+ ret i64 %v
+}
+
+!1 = !{!"branch_weights", i32 1, i32 10, i32 1000, i32 10, i32 1000, i32 10, i32 10, i32 10}
+!2 = !{!"branch_weights", i32 1, i32 100, i32 1, i32 100, i32 1, i32 100, i32 100, i32 100}
+;.
+; CHECK: [[PROF0]] = !{!"branch_weights", i32 1000, i32 51}
+; CHECK: [[PROF1]] = !{!"branch_weights", i32 50, i32 1}
+; CHECK: [[PROF2]] = !{!"branch_weights", i32 51, i32 1000, i32 1000}
+; CHECK: [[PROF3]] = !{!"branch_weights", i32 1, i32 100, i32 1, i32 100, i32 1, i32 100, i32 100, i32 100}
+;.
+; OFF: [[PROF0]] = !{!"branch_weights", i32 1, i32 10, i32 1000, i32 10, i32 10, i32 10, i32 10}
+; OFF: [[PROF1]] = !{!"branch_weights", i32 1, i32 10, i32 1000, i32 10, i32 1000, i32 10, i32 10, i32 10}
+; OFF: [[PROF2]] = !{!"branch_weights", i32 1, i32 100, i32 1, i32 100, i32 1, i32 100, i32 100, i32 100}
+;.
>From 578ddf581cb99475a9b849e6460c3196f6bf2214 Mon Sep 17 00:00:00 2001
From: Nazar Mokrynskyi <nazar at mokrynskyi.com>
Date: Sun, 13 Sep 2026 14:07:48 +0000
Subject: [PATCH 2/2] [ConstantMerge] Merge constants that differ only in
poison elements
Poison may be replaced by any value, so two constant arrays that agree
wherever both are defined describe the same thing: unifying them into the
more defined of the two refines both, and needs no reasoning about which
elements are read.
Lookup tables built for copies of an inlined switch come out this shape
when each copy tests a different case ahead of its table, leaving a slot
in each that nothing loads. The tables are then alike everywhere but in
the slots one copy left poison and another filled in, which is enough to
keep them apart and to pay for a whole table, and its relocations, per
copy.
This cannot use the hash map the identical-initializer merging above
relies on, since constants that unify need not be equal and no hash of
one finds the other: they disagree exactly where a hash would read.
Only a constant holding poison has anything to gain, so those drive the
search, each compared against the constants of its type. It merges into
the one needing fewest elements filled in, so the result does not depend
on the order the globals appear in, and the search is bounded by
-constmerge-max-poison-candidates.
---
llvm/lib/Transforms/IPO/ConstantMerge.cpp | 124 ++++++++++++++++++
.../ConstantMerge/merge-poison-compatible.ll | 62 +++++++++
2 files changed, 186 insertions(+)
create mode 100644 llvm/test/Transforms/ConstantMerge/merge-poison-compatible.ll
diff --git a/llvm/lib/Transforms/IPO/ConstantMerge.cpp b/llvm/lib/Transforms/IPO/ConstantMerge.cpp
index 480f4c5f68f79..2bfc46478300e 100644
--- a/llvm/lib/Transforms/IPO/ConstantMerge.cpp
+++ b/llvm/lib/Transforms/IPO/ConstantMerge.cpp
@@ -18,6 +18,7 @@
#include "llvm/Transforms/IPO/ConstantMerge.h"
#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/MapVector.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
#include "llvm/ADT/Statistic.h"
@@ -29,6 +30,7 @@
#include "llvm/IR/LLVMContext.h"
#include "llvm/IR/Module.h"
#include "llvm/Support/Casting.h"
+#include "llvm/Support/CommandLine.h"
#include "llvm/Support/Debug.h"
#include "llvm/Transforms/IPO.h"
#include <algorithm>
@@ -39,7 +41,14 @@ using namespace llvm;
#define DEBUG_TYPE "constmerge"
+static cl::opt<unsigned> MaxPoisonCandidates(
+ "constmerge-max-poison-candidates", cl::Hidden, cl::init(1024),
+ cl::desc("Largest set of constants of one type that a constant holding "
+ "poison is compared against. A larger set is left alone"));
+
STATISTIC(NumIdenticalMerged, "Number of identical global constants merged");
+STATISTIC(NumPoisonMerged,
+ "Number of global constants merged that differed only in poison");
/// Find values that are marked as llvm.used.
static void FindUsedValues(GlobalVariable *LLVMUsed,
@@ -135,6 +144,119 @@ static void replace(Module &M, GlobalVariable *Old, GlobalVariable *New) {
Old->eraseFromParent();
}
+/// Unify two array constants that differ only where one of them holds poison.
+///
+/// Poison may be replaced by any value, so a table holding poison at an index
+/// and a table holding a defined value there describe the same thing: taking
+/// the defined value refines both. Returns the unified constant, or null if the
+/// two disagree anywhere they are both defined.
+///
+/// Copies of an inlined switch produce exactly this shape when each copy tests
+/// a different case ahead of its lookup table: every copy's table is the same
+/// but for the slots standing in for the cases that copy tested, which nothing
+/// loads and which are therefore poison.
+static Constant *unifyPoison(Constant *A, Constant *B) {
+ if (A == B)
+ return A;
+ auto *ATy = dyn_cast<ArrayType>(A->getType());
+ if (!ATy || A->getType() != B->getType())
+ return nullptr;
+
+ uint64_t N = ATy->getNumElements();
+ SmallVector<Constant *, 64> Unified(N);
+ for (uint64_t I = 0; I != N; ++I) {
+ Constant *EA = A->getAggregateElement(I);
+ Constant *EB = B->getAggregateElement(I);
+ if (!EA || !EB)
+ return nullptr;
+ if (EA == EB) {
+ Unified[I] = EA;
+ continue;
+ }
+ if (isa<PoisonValue>(EA)) {
+ Unified[I] = EB;
+ continue;
+ }
+ if (isa<PoisonValue>(EB)) {
+ Unified[I] = EA;
+ continue;
+ }
+ return nullptr;
+ }
+ return ConstantArray::get(ATy, Unified);
+}
+
+/// Merge globals whose initializers differ only in poison elements.
+///
+/// Kept apart from the identical-initializer merging above because it cannot
+/// use a hash map. Two constants that unify need not be equal, and no hash of
+/// one of them can find the other, since they disagree exactly where a hash
+/// would read. Only a constant that holds poison has anything to gain, so those
+/// drive the search, and each is compared against the constants of its type,
+/// bounded by MaxPoisonCandidates. Giving that bound up costs an optimisation,
+/// never correctness.
+static size_t
+mergePoisonCompatible(Module &M,
+ const SmallPtrSetImpl<const GlobalValue *> &UsedGlobals) {
+ // Candidates, grouped by type so only plausible pairs are compared.
+ MapVector<Type *, SmallVector<GlobalVariable *, 8>> ByType;
+ SmallVector<GlobalVariable *, 8> HoldsPoison;
+ for (GlobalVariable &GV : M.globals()) {
+ // The same conditions the merging above puts on a constant before it may
+ // stand in for another. A global it refuses to make canonical must not
+ // become one here either, since replace() assumes it could have.
+ if (isUnmergeableGlobal(&GV, UsedGlobals) || !GV.hasLocalLinkage() ||
+ GV.isWeakForLinker() || GV.hasMetadataOtherThanDebugLocAndGuid())
+ continue;
+ // Only ConstantArray can hold poison. An array of defined integers is a
+ // ConstantDataArray, which nothing here can unify with anyway.
+ auto *Init = dyn_cast<ConstantArray>(GV.getInitializer());
+ if (!Init)
+ continue;
+ ByType[GV.getValueType()].push_back(&GV);
+ if (any_of(Init->operands(),
+ [](const Use &U) { return isa<PoisonValue>(U.get()); }))
+ HoldsPoison.push_back(&GV);
+ }
+
+ size_t Merged = 0;
+ for (GlobalVariable *GV : HoldsPoison) {
+ SmallVectorImpl<GlobalVariable *> &Candidates = ByType[GV->getValueType()];
+ if (Candidates.size() > MaxPoisonCandidates)
+ continue;
+
+ // Merge into the constant that needs the fewest elements filled in, so that
+ // the outcome does not depend on the order the globals happen to appear in.
+ // Ties go to the earlier one, which keeps it deterministic.
+ GlobalVariable *Into = nullptr;
+ Constant *Unified = nullptr;
+ unsigned FewestPoison = 0;
+ for (GlobalVariable *C : Candidates) {
+ if (C == GV || C->getParent() != &M)
+ continue;
+ Constant *U = unifyPoison(C->getInitializer(), GV->getInitializer());
+ if (!U)
+ continue;
+ unsigned Poison =
+ count_if(cast<ConstantArray>(C->getInitializer())->operands(),
+ [](const Use &Op) { return isa<PoisonValue>(Op.get()); });
+ if (Into && Poison >= FewestPoison)
+ continue;
+ Into = C;
+ Unified = U;
+ FewestPoison = Poison;
+ }
+ if (!Into || makeMergeable(GV, Into) == CanMerge::No)
+ continue;
+
+ Into->setInitializer(Unified);
+ replace(M, GV, Into);
+ ++Merged;
+ ++NumPoisonMerged;
+ }
+ return Merged;
+}
+
static bool mergeConstants(Module &M) {
// Find all the globals that are marked "used". These cannot be merged.
SmallPtrSet<const GlobalValue*, 8> UsedGlobals;
@@ -244,6 +366,8 @@ static bool mergeConstants(Module &M) {
CMap.clear();
}
+ ChangesMade += mergePoisonCompatible(M, UsedGlobals);
+
return ChangesMade;
}
diff --git a/llvm/test/Transforms/ConstantMerge/merge-poison-compatible.ll b/llvm/test/Transforms/ConstantMerge/merge-poison-compatible.ll
new file mode 100644
index 0000000000000..204b4eea11872
--- /dev/null
+++ b/llvm/test/Transforms/ConstantMerge/merge-poison-compatible.ll
@@ -0,0 +1,62 @@
+; Constant arrays that differ only where one of them holds poison describe the
+; same value, so they can be merged into the more defined of the two. Lookup
+; tables built for copies of an inlined switch differ exactly this way.
+; RUN: opt -passes=constmerge -S < %s | FileCheck %s
+; Comparing candidates against each other is bounded. With no room for it,
+; nothing is merged and nothing is wrong.
+; RUN: opt -passes=constmerge -constmerge-max-poison-candidates=1 -S < %s \
+; RUN: | FileCheck %s --check-prefix=NOBUDGET
+
+; NOBUDGET: @a = private unnamed_addr constant [4 x ptr] [ptr @f0, ptr poison, ptr @f2, ptr @f3]
+; NOBUDGET: @b = private unnamed_addr constant [4 x ptr] [ptr @f0, ptr @f1, ptr poison, ptr @f3]
+; NOBUDGET: @c = private unnamed_addr constant [4 x ptr] [ptr @f0, ptr @f1, ptr @f2, ptr @f3]
+
+; The merge target is the constant needing fewest elements filled in, not
+; whichever comes first, so reordering the globals does not change the result.
+
+; CHECK: @c = private unnamed_addr constant [4 x ptr] [ptr @f0, ptr @f1, ptr @f2, ptr @f3]
+; CHECK-NOT: @a =
+; CHECK-NOT: @b =
+; CHECK: @different = private unnamed_addr constant [4 x ptr] [ptr @f3, ptr @f1, ptr @f2, ptr @f3]
+
+ at a = private unnamed_addr constant [4 x ptr] [ptr @f0, ptr poison, ptr @f2, ptr @f3]
+ at b = private unnamed_addr constant [4 x ptr] [ptr @f0, ptr @f1, ptr poison, ptr @f3]
+ at c = private unnamed_addr constant [4 x ptr] [ptr @f0, ptr @f1, ptr @f2, ptr @f3]
+ at different = private unnamed_addr constant [4 x ptr] [ptr @f3, ptr @f1, ptr @f2, ptr @f3]
+
+declare void @f0()
+declare void @f1()
+declare void @f2()
+declare void @f3()
+
+define ptr @ua(i64 %i) {
+; CHECK-LABEL: @ua(
+; CHECK: getelementptr [4 x ptr], ptr @c,
+ %p = getelementptr [4 x ptr], ptr @a, i64 0, i64 %i
+ %v = load ptr, ptr %p
+ ret ptr %v
+}
+
+define ptr @ub(i64 %i) {
+; CHECK-LABEL: @ub(
+; CHECK: getelementptr [4 x ptr], ptr @c,
+ %p = getelementptr [4 x ptr], ptr @b, i64 0, i64 %i
+ %v = load ptr, ptr %p
+ ret ptr %v
+}
+
+define ptr @uc(i64 %i) {
+; CHECK-LABEL: @uc(
+; CHECK: getelementptr [4 x ptr], ptr @c,
+ %p = getelementptr [4 x ptr], ptr @c, i64 0, i64 %i
+ %v = load ptr, ptr %p
+ ret ptr %v
+}
+
+define ptr @ud(i64 %i) {
+; CHECK-LABEL: @ud(
+; CHECK: getelementptr [4 x ptr], ptr @different,
+ %p = getelementptr [4 x ptr], ptr @different, i64 0, i64 %i
+ %v = load ptr, ptr %p
+ ret ptr %v
+}
More information about the llvm-commits
mailing list