[llvm] [LCSSA] Preserve MemorySSA when removing lifetime markers (PR #221690)
Karim Alweheshy via llvm-commits
llvm-commits at lists.llvm.org
Mon Sep 7 02:36:16 PDT 2026
https://github.com/karim-alweheshy created https://github.com/llvm/llvm-project/pull/221690
## Summary
Follow up on #210811 by preserving MemorySSA when LCSSA removes lifetime
markers. The previous change keeps lifetime intrinsics from receiving invalid
PHI operands, but deleting those instructions without removing their
MemoryAccess records leaves a cached MemorySSA analysis inconsistent.
LCSSA currently advertises that it preserves that analysis, and other
MemorySSA-preserving transforms also call its utility entry points.
Thread an optional MemorySSAUpdater through the LCSSA helpers and callers that
preserve the analysis. Remove each marker's memory access before erasing the
instruction. Forward the updater through SCEVExpander's LCSSA repair as well,
including the indirect SimplifyIndVar caller. Callers that invalidate MemorySSA
can keep using the default null argument. Include LoopVersioning's newly
MemorySSA-preserving direct helper and two SCEV expanders on current main.
Compiled clients must rebuild for the
changed helper signatures and expander layout.
This keeps the existing lifetime-marker behavior and analysis preservation;
it does not disable loop optimization or force analysis reconstruction.
## Regression coverage
- Extend the existing lifetime-marker test with
`function(require<memoryssa>,lcssa,verify<memoryssa>)`. The unpatched
implementation fails MemorySSA verification.
- Add a pointer-expansion unit test checking the exit PHI, removal of both
lifetime markers, reconnection of surviving store/load accesses, and
IR/LCSSA/full-MemorySSA validity.
- Removing only the expander's updater forwarding causes the new unit test
to fail its memory-chain/access-count checks and MemorySSA verification.
## Validation
The patch was built and tested in an assert-enabled LLVM 24 AArch64 build:
- All 24 LCSSA tests pass.
- Across the ten affected transform suites: 784 pass, 155 are unsupported by
that target configuration, and three are expected failures.
- All 183 Transform Utils unit tests pass.
The branch is based on main at `1191f353cefe3c7ceb8e76fd03296e082a247685`.
The local executable tests above used the retained earlier LLVM 24 build;
the additional LoopVersioning forwarding is source-reviewed, not covered by
that older build. This is not a claim of a complete current-main build or
all-target test coverage. Further validation is in progress.
>From c77b250d76eed8e02489bdb540dbea14935ac8b8 Mon Sep 17 00:00:00 2001
From: Karim Alweheshy <karim.alweheshy at reddit.com>
Date: Mon, 7 Sep 2026 11:32:19 +0200
Subject: [PATCH] [LCSSA] Preserve MemorySSA when removing lifetime markers
Thread the existing MemorySSA updater through LCSSA helpers and preserving callers, including SCEV expansion and loop versioning. Remove memory accesses before deleting lifetime instructions. Add cached-analysis and expander regressions. Follow up on #210811.
---
.../include/llvm/Transforms/Utils/LoopUtils.h | 14 +++-
.../Utils/ScalarEvolutionExpander.h | 13 ++-
llvm/lib/Transforms/Scalar/IndVarSimplify.cpp | 2 +-
llvm/lib/Transforms/Scalar/LICM.cpp | 2 +-
llvm/lib/Transforms/Scalar/LoopFlatten.cpp | 7 +-
.../Transforms/Scalar/LoopIdiomRecognize.cpp | 8 +-
.../lib/Transforms/Scalar/LoopPredication.cpp | 2 +-
.../lib/Transforms/Scalar/LoopSimplifyCFG.cpp | 2 +-
.../Transforms/Scalar/LoopStrengthReduce.cpp | 8 +-
llvm/lib/Transforms/Scalar/LoopTermFold.cpp | 2 +-
.../Transforms/Scalar/SimpleLoopUnswitch.cpp | 4 +-
llvm/lib/Transforms/Utils/LCSSA.cpp | 62 ++++++++------
llvm/lib/Transforms/Utils/LoopSimplify.cpp | 2 +-
llvm/lib/Transforms/Utils/LoopUtils.cpp | 2 +-
llvm/lib/Transforms/Utils/LoopVersioning.cpp | 6 +-
.../Utils/ScalarEvolutionExpander.cpp | 2 +-
llvm/lib/Transforms/Utils/SimplifyIndVar.cpp | 3 +-
.../test/Transforms/LCSSA/lifetime-markers.ll | 1 +
.../Utils/ScalarEvolutionExpanderTest.cpp | 80 +++++++++++++++++++
19 files changed, 168 insertions(+), 54 deletions(-)
diff --git a/llvm/include/llvm/Transforms/Utils/LoopUtils.h b/llvm/include/llvm/Transforms/Utils/LoopUtils.h
index 74c549be35ddf..3cbbc54ff6500 100644
--- a/llvm/include/llvm/Transforms/Utils/LoopUtils.h
+++ b/llvm/include/llvm/Transforms/Utils/LoopUtils.h
@@ -83,12 +83,15 @@ LLVM_ABI bool formDedicatedExitBlocks(Loop *L, DominatorTree *DT, LoopInfo *LI,
///
/// If \p InsertedPHIs is not nullptr, inserted phis will be added to this
/// vector.
+///
+/// Pass \p MSSAU when preserving MemorySSA: lifetime markers may be removed.
LLVM_ABI bool
formLCSSAForInstructions(SmallVectorImpl<Instruction *> &Worklist,
const DominatorTree &DT, const LoopInfo &LI,
ScalarEvolution *SE,
SmallVectorImpl<PHINode *> *PHIsToRemove = nullptr,
- SmallVectorImpl<PHINode *> *InsertedPHIs = nullptr);
+ SmallVectorImpl<PHINode *> *InsertedPHIs = nullptr,
+ MemorySSAUpdater *MSSAU = nullptr);
/// Put loop into LCSSA form.
///
@@ -101,9 +104,11 @@ formLCSSAForInstructions(SmallVectorImpl<Instruction *> &Worklist,
///
/// If ScalarEvolution is passed in, it will be preserved.
///
+/// Pass \p MSSAU when preserving MemorySSA: lifetime markers may be removed.
+///
/// Returns true if any modifications are made to the loop.
LLVM_ABI bool formLCSSA(Loop &L, const DominatorTree &DT, const LoopInfo *LI,
- ScalarEvolution *SE);
+ ScalarEvolution *SE, MemorySSAUpdater *MSSAU = nullptr);
/// Put a loop nest into LCSSA form.
///
@@ -113,9 +118,12 @@ LLVM_ABI bool formLCSSA(Loop &L, const DominatorTree &DT, const LoopInfo *LI,
///
/// If ScalarEvolution is passed in, it will be preserved.
///
+/// Pass \p MSSAU when preserving MemorySSA: lifetime markers may be removed.
+///
/// Returns true if any modifications are made to the loop.
LLVM_ABI bool formLCSSARecursively(Loop &L, const DominatorTree &DT,
- const LoopInfo *LI, ScalarEvolution *SE);
+ const LoopInfo *LI, ScalarEvolution *SE,
+ MemorySSAUpdater *MSSAU = nullptr);
/// Flags controlling how much is checked when sinking or hoisting
/// instructions. The number of memory access in the loop (and whether there
diff --git a/llvm/include/llvm/Transforms/Utils/ScalarEvolutionExpander.h b/llvm/include/llvm/Transforms/Utils/ScalarEvolutionExpander.h
index c98c0cb52fa9c..3d7839d411812 100644
--- a/llvm/include/llvm/Transforms/Utils/ScalarEvolutionExpander.h
+++ b/llvm/include/llvm/Transforms/Utils/ScalarEvolutionExpander.h
@@ -28,6 +28,8 @@
#include "llvm/Support/InstructionCost.h"
namespace llvm {
+class MemorySSAUpdater;
+
LLVM_ABI extern cl::opt<unsigned> SCEVCheapExpansionBudget;
/// struct for holding enough information to help calculate the cost of the
@@ -74,6 +76,9 @@ class SCEVExpander : public SCEVUseVisitor<SCEVExpander, Value *> {
/// Indicates whether LCSSA phis should be created for inserted values.
bool PreserveLCSSA;
+ /// Updated when LCSSA repair removes lifetime markers.
+ MemorySSAUpdater *MSSAU;
+
// InsertedExpressions caches Values for reuse, so must track RAUW.
DenseMap<std::pair<SCEVUse, Instruction *>, TrackingVH<Value>>
InsertedExpressions;
@@ -189,10 +194,12 @@ class SCEVExpander : public SCEVUseVisitor<SCEVExpander, Value *> {
public:
/// Construct a SCEVExpander in "canonical" mode.
+ /// Pass \p MSSAU when preserving MemorySSA during LCSSA repair.
explicit SCEVExpander(ScalarEvolution &SE, const char *Name,
- bool PreserveLCSSA = true)
+ bool PreserveLCSSA = true,
+ MemorySSAUpdater *MSSAU = nullptr)
: SE(SE), DL(SE.getDataLayout()), IVName(Name),
- PreserveLCSSA(PreserveLCSSA), IVIncInsertLoop(nullptr),
+ PreserveLCSSA(PreserveLCSSA), MSSAU(MSSAU), IVIncInsertLoop(nullptr),
IVIncInsertPos(nullptr), CanonicalMode(true), LSRMode(false),
Builder(SE.getContext(), InstSimplifyFolder(DL),
IRBuilderCallbackInserter(
@@ -207,6 +214,8 @@ class SCEVExpander : public SCEVUseVisitor<SCEVExpander, Value *> {
assert(InsertPointGuards.empty());
}
+ MemorySSAUpdater *getMemorySSAUpdater() const { return MSSAU; }
+
#if LLVM_ENABLE_ABI_BREAKING_CHECKS
void setDebugType(const char *s) { DebugType = s; }
#endif
diff --git a/llvm/lib/Transforms/Scalar/IndVarSimplify.cpp b/llvm/lib/Transforms/Scalar/IndVarSimplify.cpp
index 4b1fb39ce9932..f2d062892c270 100644
--- a/llvm/lib/Transforms/Scalar/IndVarSimplify.cpp
+++ b/llvm/lib/Transforms/Scalar/IndVarSimplify.cpp
@@ -2077,7 +2077,7 @@ bool IndVarSimplify::run(Loop *L) {
Changed |= rewriteNonIntegerIVs(L);
// Create a rewriter object which we'll use to transform the code with.
- SCEVExpander Rewriter(*SE, "indvars");
+ SCEVExpander Rewriter(*SE, "indvars", true, MSSAU.get());
#if LLVM_ENABLE_ABI_BREAKING_CHECKS
Rewriter.setDebugType(DEBUG_TYPE);
#endif
diff --git a/llvm/lib/Transforms/Scalar/LICM.cpp b/llvm/lib/Transforms/Scalar/LICM.cpp
index ee4c7603fdd2d..44e60450fab7e 100644
--- a/llvm/lib/Transforms/Scalar/LICM.cpp
+++ b/llvm/lib/Transforms/Scalar/LICM.cpp
@@ -538,7 +538,7 @@ bool LoopInvariantCodeMotion::runOnLoop(Loop *L, AAResults *AA, LoopInfo *LI,
// SSAUpdater strategy during promotion that was LCSSA aware and reformed
// it as it went.
if (Promoted)
- formLCSSARecursively(*L, *DT, LI, SE);
+ formLCSSARecursively(*L, *DT, LI, SE, &MSSAU);
Changed |= Promoted;
}
diff --git a/llvm/lib/Transforms/Scalar/LoopFlatten.cpp b/llvm/lib/Transforms/Scalar/LoopFlatten.cpp
index 6d81fa935b491..3fa9a5f150744 100644
--- a/llvm/lib/Transforms/Scalar/LoopFlatten.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopFlatten.cpp
@@ -837,7 +837,8 @@ static bool DoFlattenLoopPair(FlattenInfo &FI, DominatorTree *DT, LoopInfo *LI,
static bool CanWidenIV(FlattenInfo &FI, DominatorTree *DT, LoopInfo *LI,
ScalarEvolution *SE, AssumptionCache *AC,
- const TargetTransformInfo *TTI) {
+ const TargetTransformInfo *TTI,
+ MemorySSAUpdater *MSSAU) {
if (!WidenIV) {
LLVM_DEBUG(dbgs() << "Widening the IVs is disabled\n");
return false;
@@ -862,7 +863,7 @@ static bool CanWidenIV(FlattenInfo &FI, DominatorTree *DT, LoopInfo *LI,
return false;
}
- SCEVExpander Rewriter(*SE, "loopflatten");
+ SCEVExpander Rewriter(*SE, "loopflatten", true, MSSAU);
SmallVector<WeakTrackingVH, 4> DeadInsts;
unsigned ElimExt = 0;
unsigned Widened = 0;
@@ -916,7 +917,7 @@ static bool FlattenLoopPair(FlattenInfo &FI, DominatorTree *DT, LoopInfo *LI,
return false;
// Check if we can widen the induction variables to avoid overflow checks.
- bool CanFlatten = CanWidenIV(FI, DT, LI, SE, AC, TTI);
+ bool CanFlatten = CanWidenIV(FI, DT, LI, SE, AC, TTI, MSSAU);
// It can happen that after widening of the IV, flattening may not be
// possible/happening, e.g. when it is deemed unprofitable. So bail here if
diff --git a/llvm/lib/Transforms/Scalar/LoopIdiomRecognize.cpp b/llvm/lib/Transforms/Scalar/LoopIdiomRecognize.cpp
index 4e5cd8462cc4d..f4cdbb4c99ad0 100644
--- a/llvm/lib/Transforms/Scalar/LoopIdiomRecognize.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopIdiomRecognize.cpp
@@ -1107,7 +1107,7 @@ bool LoopIdiomRecognize::processLoopStridedStore(
unsigned DestAS = DestPtr->getType()->getPointerAddressSpace();
BasicBlock *Preheader = CurLoop->getLoopPreheader();
IRBuilder<> Builder(Preheader->getTerminator());
- SCEVExpander Expander(*SE, "loop-idiom");
+ SCEVExpander Expander(*SE, "loop-idiom", true, MSSAU.get());
SCEVExpanderCleaner ExpCleaner(Expander);
Type *DestInt8PtrTy = Builder.getPtrTy(DestAS);
@@ -1367,7 +1367,7 @@ bool LoopIdiomRecognize::processLoopStoreOfLoopLoad(
// header. This allows us to insert code for it in the preheader.
BasicBlock *Preheader = CurLoop->getLoopPreheader();
IRBuilder<> Builder(Preheader->getTerminator());
- SCEVExpander Expander(*SE, "loop-idiom");
+ SCEVExpander Expander(*SE, "loop-idiom", true, MSSAU.get());
SCEVExpanderCleaner ExpCleaner(Expander);
@@ -2249,7 +2249,7 @@ bool LoopIdiomRecognize::recognizeAndInsertStrLen() {
IRBuilder<> Builder(Preheader->getTerminator());
Builder.SetCurrentDebugLocation(CurLoop->getStartLoc());
- SCEVExpander Expander(*SE, "strlen_idiom");
+ SCEVExpander Expander(*SE, "strlen_idiom", true, MSSAU.get());
Value *MaterialzedBase = Expander.expandCodeFor(
Verifier.LoadBaseEv, Verifier.LoadBaseEv->getType(),
Builder.GetInsertPoint());
@@ -3792,7 +3792,7 @@ bool LoopIdiomRecognize::recognizeShiftUntilZero() {
Val->getName() + ".numactivebits", /*HasNUW=*/true,
/*HasNSW=*/Bitwidth != 2);
- SCEVExpander Expander(*SE, "loop-idiom");
+ SCEVExpander Expander(*SE, "loop-idiom", true, MSSAU.get());
Expander.setInsertPoint(&*Builder.GetInsertPoint());
Value *ExtraOffset = Expander.expandCodeFor(ExtraOffsetExpr);
diff --git a/llvm/lib/Transforms/Scalar/LoopPredication.cpp b/llvm/lib/Transforms/Scalar/LoopPredication.cpp
index 2deaeeb0792e6..ffea6ce14d8b6 100644
--- a/llvm/lib/Transforms/Scalar/LoopPredication.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopPredication.cpp
@@ -1234,7 +1234,7 @@ bool LoopPredication::runOnLoop(Loop *Loop) {
cast<CondBrInst>(BB->getTerminator()));
}
- SCEVExpander Expander(*SE, "loop-predication");
+ SCEVExpander Expander(*SE, "loop-predication", true, MSSAU);
bool Changed = false;
for (auto *Guard : Guards)
Changed |= widenGuardConditions(Guard, Expander);
diff --git a/llvm/lib/Transforms/Scalar/LoopSimplifyCFG.cpp b/llvm/lib/Transforms/Scalar/LoopSimplifyCFG.cpp
index 09d5b1d4ec676..0a2f530a9c0a8 100644
--- a/llvm/lib/Transforms/Scalar/LoopSimplifyCFG.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopSimplifyCFG.cpp
@@ -437,7 +437,7 @@ class ConstantTerminatorFoldingImpl {
else
DTU.applyUpdates(DTUpdates);
DTUpdates.clear();
- formLCSSARecursively(*FixLCSSALoop, DT, &LI, &SE);
+ formLCSSARecursively(*FixLCSSALoop, DT, &LI, &SE, MSSAU);
SE.forgetBlockAndLoopDispositions();
}
}
diff --git a/llvm/lib/Transforms/Scalar/LoopStrengthReduce.cpp b/llvm/lib/Transforms/Scalar/LoopStrengthReduce.cpp
index e2ca5f166dbb5..470ab21027290 100644
--- a/llvm/lib/Transforms/Scalar/LoopStrengthReduce.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopStrengthReduce.cpp
@@ -6241,7 +6241,7 @@ void LSRInstance::ImplementSolution(
}
auto InsertedInsts = InsertedNonLCSSAInsts.takeVector();
- formLCSSAForInstructions(InsertedInsts, DT, LI, &SE);
+ formLCSSAForInstructions(InsertedInsts, DT, LI, &SE, nullptr, nullptr, MSSAU);
for (const IVChain &Chain : IVChainVec) {
GenerateIVChain(Chain, DeadInsts);
@@ -6313,8 +6313,8 @@ LSRInstance::LSRInstance(Loop *L, IVUsers &IU, ScalarEvolution &SE,
MSSAU(MSSAU), AMK(PreferredAddresingMode.getNumOccurrences() > 0
? PreferredAddresingMode
: TTI.getPreferredAddressingMode(L, &SE)),
- Rewriter(SE, "lsr", PreserveLCSSA), ShouldPreserveLCSSA(PreserveLCSSA),
- BaselineCost(L, SE, TTI, AMK) {
+ Rewriter(SE, "lsr", PreserveLCSSA, MSSAU),
+ ShouldPreserveLCSSA(PreserveLCSSA), BaselineCost(L, SE, TTI, AMK) {
// If LoopSimplify form is not available, stay out of trouble.
if (!L->isLoopSimplifyForm())
return;
@@ -7229,7 +7229,7 @@ static bool ReduceLoopStrength(Loop *L, IVUsers &IU, ScalarEvolution &SE,
// skip the updates in each loop iteration.
if (L->isRecursivelyLCSSAForm(DT, LI) && L->getExitBlock()) {
SmallVector<WeakTrackingVH, 16> DeadInsts;
- SCEVExpander Rewriter(SE, "lsr", true);
+ SCEVExpander Rewriter(SE, "lsr", true, MSSAU.get());
int Rewrites = rewriteLoopExitValues(L, &LI, &TLI, &SE, &TTI, Rewriter, &DT,
UnusedIndVarInLoop, DeadInsts);
Rewriter.clear();
diff --git a/llvm/lib/Transforms/Scalar/LoopTermFold.cpp b/llvm/lib/Transforms/Scalar/LoopTermFold.cpp
index e6eccdd7d616e..8a4baae308f74 100644
--- a/llvm/lib/Transforms/Scalar/LoopTermFold.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopTermFold.cpp
@@ -259,7 +259,7 @@ static bool RunTermFold(Loop *L, ScalarEvolution &SE, DominatorTree &DT,
cast<Instruction>(LoopValue)->dropPoisonGeneratingFlags();
// SCEVExpander for both use in preheader and latch
- SCEVExpander Expander(SE, "lsr_fold_term_cond");
+ SCEVExpander Expander(SE, "lsr_fold_term_cond", true, MSSAU.get());
assert(Expander.isSafeToExpand(TermValueS) &&
"Terminating value was checked safe in canFoldTerminatingCondition");
diff --git a/llvm/lib/Transforms/Scalar/SimpleLoopUnswitch.cpp b/llvm/lib/Transforms/Scalar/SimpleLoopUnswitch.cpp
index ba61521e25d8d..73a590c2d0826 100644
--- a/llvm/lib/Transforms/Scalar/SimpleLoopUnswitch.cpp
+++ b/llvm/lib/Transforms/Scalar/SimpleLoopUnswitch.cpp
@@ -516,7 +516,7 @@ static void hoistLoopToNewParent(Loop &L, BasicBlock &Preheader,
// Because we just hoisted a loop out of this one, we have essentially
// created new exit paths from it. That means we need to form LCSSA PHI
// nodes for values used in the no-longer-nested loop.
- formLCSSA(*OldContainingL, DT, &LI, SE);
+ formLCSSA(*OldContainingL, DT, &LI, SE, MSSAU);
// We shouldn't need to form dedicated exits because the exit introduced
// here is the (just split by unswitching) preheader. However, after trivial
@@ -2401,7 +2401,7 @@ static void unswitchNontrivialInvariants(
// First build LCSSA for this loop so that we can preserve it when
// forming dedicated exits. We don't want to perturb some other loop's
// LCSSA while doing that CFG edit.
- formLCSSA(UpdateL, DT, &LI, SE);
+ formLCSSA(UpdateL, DT, &LI, SE, MSSAU);
// For loops reached by this loop's original exit blocks we may
// introduced new, non-dedicated exits. At least try to re-form dedicated
diff --git a/llvm/lib/Transforms/Utils/LCSSA.cpp b/llvm/lib/Transforms/Utils/LCSSA.cpp
index 415c6baff20d7..1310eeea122ec 100644
--- a/llvm/lib/Transforms/Utils/LCSSA.cpp
+++ b/llvm/lib/Transforms/Utils/LCSSA.cpp
@@ -36,6 +36,7 @@
#include "llvm/Analysis/LoopInfo.h"
#include "llvm/Analysis/LoopPass.h"
#include "llvm/Analysis/MemorySSA.h"
+#include "llvm/Analysis/MemorySSAUpdater.h"
#include "llvm/Analysis/ScalarEvolution.h"
#include "llvm/Analysis/ScalarEvolutionAliasAnalysis.h"
#include "llvm/IR/DebugInfo.h"
@@ -48,6 +49,7 @@
#include "llvm/Transforms/Utils.h"
#include "llvm/Transforms/Utils/LoopUtils.h"
#include "llvm/Transforms/Utils/SSAUpdater.h"
+#include <optional>
using namespace llvm;
#define DEBUG_TYPE "lcssa"
@@ -78,13 +80,12 @@ using LoopExitBlocksTy = SmallDenseMap<Loop *, SmallVector<BasicBlock *, 1>>;
/// For every instruction from the worklist, check to see if it has any uses
/// that are outside the current loop. If so, insert LCSSA PHI nodes and
/// rewrite the uses.
-static bool
-formLCSSAForInstructionsImpl(SmallVectorImpl<Instruction *> &Worklist,
- const DominatorTree &DT, const LoopInfo &LI,
- ScalarEvolution *SE,
- SmallVectorImpl<PHINode *> *PHIsToRemove,
- SmallVectorImpl<PHINode *> *InsertedPHIs,
- LoopExitBlocksTy &LoopExitBlocks) {
+static bool formLCSSAForInstructionsImpl(
+ SmallVectorImpl<Instruction *> &Worklist, const DominatorTree &DT,
+ const LoopInfo &LI, ScalarEvolution *SE,
+ SmallVectorImpl<PHINode *> *PHIsToRemove,
+ SmallVectorImpl<PHINode *> *InsertedPHIs, LoopExitBlocksTy &LoopExitBlocks,
+ MemorySSAUpdater *MSSAU) {
SmallVector<Use *, 16> UsesToRewrite;
SmallSetVector<PHINode *, 16> LocalPHIsToRemove;
PredIteratorCache PredCache;
@@ -141,8 +142,11 @@ formLCSSAForInstructionsImpl(SmallVectorImpl<Instruction *> &Worklist,
if (DropLifetimeMarkers) {
// Use-list order is arbitrary, so wait until all markers are collected.
- for (Instruction *Marker : LifetimeMarkers)
+ for (Instruction *Marker : LifetimeMarkers) {
+ if (MSSAU)
+ MSSAU->removeMemoryAccess(Marker);
Marker->eraseFromParent();
+ }
Changed = true;
}
@@ -329,11 +333,12 @@ bool llvm::formLCSSAForInstructions(SmallVectorImpl<Instruction *> &Worklist,
const DominatorTree &DT, const LoopInfo &LI,
ScalarEvolution *SE,
SmallVectorImpl<PHINode *> *PHIsToRemove,
- SmallVectorImpl<PHINode *> *InsertedPHIs) {
+ SmallVectorImpl<PHINode *> *InsertedPHIs,
+ MemorySSAUpdater *MSSAU) {
LoopExitBlocksTy LoopExitBlocks;
return formLCSSAForInstructionsImpl(Worklist, DT, LI, SE, PHIsToRemove,
- InsertedPHIs, LoopExitBlocks);
+ InsertedPHIs, LoopExitBlocks, MSSAU);
}
// Compute the set of BasicBlocks in the loop `L` dominating at least one exit.
@@ -380,8 +385,8 @@ static void computeBlocksDominatingExits(
}
static bool formLCSSAImpl(Loop &L, const DominatorTree &DT, const LoopInfo *LI,
- ScalarEvolution *SE,
- LoopExitBlocksTy &LoopExitBlocks) {
+ ScalarEvolution *SE, LoopExitBlocksTy &LoopExitBlocks,
+ MemorySSAUpdater *MSSAU) {
bool Changed = false;
#ifdef EXPENSIVE_CHECKS
@@ -437,7 +442,7 @@ static bool formLCSSAImpl(Loop &L, const DominatorTree &DT, const LoopInfo *LI,
}
Changed = formLCSSAForInstructionsImpl(Worklist, DT, *LI, SE, nullptr,
- nullptr, LoopExitBlocks);
+ nullptr, LoopExitBlocks, MSSAU);
assert(L.isLCSSAForm(DT));
@@ -445,40 +450,43 @@ static bool formLCSSAImpl(Loop &L, const DominatorTree &DT, const LoopInfo *LI,
}
bool llvm::formLCSSA(Loop &L, const DominatorTree &DT, const LoopInfo *LI,
- ScalarEvolution *SE) {
+ ScalarEvolution *SE, MemorySSAUpdater *MSSAU) {
LoopExitBlocksTy LoopExitBlocks;
- return formLCSSAImpl(L, DT, LI, SE, LoopExitBlocks);
+ return formLCSSAImpl(L, DT, LI, SE, LoopExitBlocks, MSSAU);
}
/// Process a loop nest depth first.
static bool formLCSSARecursivelyImpl(Loop &L, const DominatorTree &DT,
const LoopInfo *LI, ScalarEvolution *SE,
- LoopExitBlocksTy &LoopExitBlocks) {
+ LoopExitBlocksTy &LoopExitBlocks,
+ MemorySSAUpdater *MSSAU) {
bool Changed = false;
// Recurse depth-first through inner loops.
for (Loop *SubLoop : L.getSubLoops())
- Changed |= formLCSSARecursivelyImpl(*SubLoop, DT, LI, SE, LoopExitBlocks);
+ Changed |=
+ formLCSSARecursivelyImpl(*SubLoop, DT, LI, SE, LoopExitBlocks, MSSAU);
- Changed |= formLCSSAImpl(L, DT, LI, SE, LoopExitBlocks);
+ Changed |= formLCSSAImpl(L, DT, LI, SE, LoopExitBlocks, MSSAU);
return Changed;
}
/// Process a loop nest depth first.
bool llvm::formLCSSARecursively(Loop &L, const DominatorTree &DT,
- const LoopInfo *LI, ScalarEvolution *SE) {
+ const LoopInfo *LI, ScalarEvolution *SE,
+ MemorySSAUpdater *MSSAU) {
LoopExitBlocksTy LoopExitBlocks;
- return formLCSSARecursivelyImpl(L, DT, LI, SE, LoopExitBlocks);
+ return formLCSSARecursivelyImpl(L, DT, LI, SE, LoopExitBlocks, MSSAU);
}
/// Process all loops in the function, inner-most out.
static bool formLCSSAOnAllLoops(const LoopInfo *LI, const DominatorTree &DT,
- ScalarEvolution *SE) {
+ ScalarEvolution *SE, MemorySSAUpdater *MSSAU) {
bool Changed = false;
for (const auto &L : *LI)
- Changed |= formLCSSARecursively(*L, DT, LI, SE);
+ Changed |= formLCSSARecursively(*L, DT, LI, SE, MSSAU);
return Changed;
}
@@ -551,14 +559,20 @@ bool LCSSAWrapperPass::runOnFunction(Function &F) {
auto *SEWP = getAnalysisIfAvailable<ScalarEvolutionWrapperPass>();
SE = SEWP ? &SEWP->getSE() : nullptr;
- return formLCSSAOnAllLoops(LI, *DT, SE);
+ std::optional<MemorySSAUpdater> MSSAU;
+ if (auto *MSSA = getAnalysisIfAvailable<MemorySSAWrapperPass>())
+ MSSAU.emplace(&MSSA->getMSSA());
+ return formLCSSAOnAllLoops(LI, *DT, SE, MSSAU ? &*MSSAU : nullptr);
}
PreservedAnalyses LCSSAPass::run(Function &F, FunctionAnalysisManager &AM) {
auto &LI = AM.getResult<LoopAnalysis>(F);
auto &DT = AM.getResult<DominatorTreeAnalysis>(F);
auto *SE = AM.getCachedResult<ScalarEvolutionAnalysis>(F);
- if (!formLCSSAOnAllLoops(&LI, DT, SE))
+ std::optional<MemorySSAUpdater> MSSAU;
+ if (auto *MSSA = AM.getCachedResult<MemorySSAAnalysis>(F))
+ MSSAU.emplace(&MSSA->getMSSA());
+ if (!formLCSSAOnAllLoops(&LI, DT, SE, MSSAU ? &*MSSAU : nullptr))
return PreservedAnalyses::all();
PreservedAnalyses PA;
diff --git a/llvm/lib/Transforms/Utils/LoopSimplify.cpp b/llvm/lib/Transforms/Utils/LoopSimplify.cpp
index ed69b7e9a27eb..7d9d85f2ccb36 100644
--- a/llvm/lib/Transforms/Utils/LoopSimplify.cpp
+++ b/llvm/lib/Transforms/Utils/LoopSimplify.cpp
@@ -334,7 +334,7 @@ static Loop *separateNestedLoop(Loop *L, BasicBlock *Preheader,
// We don't need to form LCSSA recursively, because there cannot be uses
// inside a newly created loop of defs from inner loops as those would
// already be a use of an LCSSA phi node.
- formLCSSA(*L, *DT, LI, SE);
+ formLCSSA(*L, *DT, LI, SE, MSSAU);
assert(NewOuter->isRecursivelyLCSSAForm(*DT, *LI) &&
"LCSSA is broken after separating nested loops!");
diff --git a/llvm/lib/Transforms/Utils/LoopUtils.cpp b/llvm/lib/Transforms/Utils/LoopUtils.cpp
index d3f2f0beacc6a..3faa715b09ad2 100644
--- a/llvm/lib/Transforms/Utils/LoopUtils.cpp
+++ b/llvm/lib/Transforms/Utils/LoopUtils.cpp
@@ -817,7 +817,7 @@ void llvm::breakLoopBackedge(Loop *L, DominatorTree &DT, ScalarEvolution &SE,
// exit blocks. If that happened, we need to rebuild LCSSA on the outermost
// loop which might have a had a block removed.
if (OutermostLoop != L)
- formLCSSARecursively(*OutermostLoop, DT, &LI, &SE);
+ formLCSSARecursively(*OutermostLoop, DT, &LI, &SE, MSSAU.get());
}
diff --git a/llvm/lib/Transforms/Utils/LoopVersioning.cpp b/llvm/lib/Transforms/Utils/LoopVersioning.cpp
index f77c4918d15bd..ac70c55cea0b7 100644
--- a/llvm/lib/Transforms/Utils/LoopVersioning.cpp
+++ b/llvm/lib/Transforms/Utils/LoopVersioning.cpp
@@ -69,11 +69,11 @@ void LoopVersioning::versionLoop(
BasicBlock *RuntimeCheckBB = VersionedLoop->getLoopPreheader();
const auto &RtPtrChecking = *LAI.getRuntimePointerChecking();
- SCEVExpander Exp2(*RtPtrChecking.getSE(), "induction");
+ SCEVExpander Exp2(*RtPtrChecking.getSE(), "induction", true, MSSAU);
MemRuntimeCheck = addRuntimeChecks(RuntimeCheckBB->getTerminator(),
VersionedLoop, AliasChecks, Exp2);
- SCEVExpander Exp(*SE, "scev.check");
+ SCEVExpander Exp(*SE, "scev.check", true, MSSAU);
SCEVRuntimeCheck =
Exp.expandCodeForPredicate(&Preds, RuntimeCheckBB->getTerminator());
@@ -320,7 +320,7 @@ bool runImpl(LoopInfo *LI, LoopAccessInfoManager &LAIs, DominatorTree *DT,
!LAI.getPSE().getPredicate().isAlwaysTrue())) {
// Forming LCSSA is a precondition of versioning.
if (!L->isRecursivelyLCSSAForm(*DT, *LI))
- formLCSSARecursively(*L, *DT, LI, SE);
+ formLCSSARecursively(*L, *DT, LI, SE, MSSAU);
LoopVersioning LVer(LAI, LAI.getRuntimePointerChecking()->getChecks(), L,
LI, DT, SE, MSSAU);
diff --git a/llvm/lib/Transforms/Utils/ScalarEvolutionExpander.cpp b/llvm/lib/Transforms/Utils/ScalarEvolutionExpander.cpp
index f01a674609325..7a6407ffac530 100644
--- a/llvm/lib/Transforms/Utils/ScalarEvolutionExpander.cpp
+++ b/llvm/lib/Transforms/Utils/ScalarEvolutionExpander.cpp
@@ -2493,7 +2493,7 @@ Value *SCEVExpander::fixupLCSSAFormFor(Value *V) {
SmallVector<PHINode *, 16> PHIsToRemove;
SmallVector<PHINode *, 16> InsertedPHIs;
formLCSSAForInstructions(ToUpdate, SE.DT, SE.LI, &SE, &PHIsToRemove,
- &InsertedPHIs);
+ &InsertedPHIs, MSSAU);
for (PHINode *PN : InsertedPHIs)
rememberInstruction(PN);
for (PHINode *PN : PHIsToRemove) {
diff --git a/llvm/lib/Transforms/Utils/SimplifyIndVar.cpp b/llvm/lib/Transforms/Utils/SimplifyIndVar.cpp
index 38b9a5b09d32b..dd937c56c535c 100644
--- a/llvm/lib/Transforms/Utils/SimplifyIndVar.cpp
+++ b/llvm/lib/Transforms/Utils/SimplifyIndVar.cpp
@@ -680,7 +680,8 @@ bool SimplifyIndvar::replaceIVUserWithLoopInvariant(Instruction *I) {
if (NeedToEmitLCSSAPhis) {
SmallVector<Instruction *, 1> NeedsLCSSAPhis;
NeedsLCSSAPhis.push_back(cast<Instruction>(Invariant));
- formLCSSAForInstructions(NeedsLCSSAPhis, *DT, *LI, SE);
+ formLCSSAForInstructions(NeedsLCSSAPhis, *DT, *LI, SE, nullptr, nullptr,
+ Rewriter.getMemorySSAUpdater());
LLVM_DEBUG(dbgs() << " INDVARS: Replacement breaks LCSSA form"
<< " inserting LCSSA Phis" << '\n');
}
diff --git a/llvm/test/Transforms/LCSSA/lifetime-markers.ll b/llvm/test/Transforms/LCSSA/lifetime-markers.ll
index e3c558a09b356..42ab3c29be903 100644
--- a/llvm/test/Transforms/LCSSA/lifetime-markers.ll
+++ b/llvm/test/Transforms/LCSSA/lifetime-markers.ll
@@ -1,5 +1,6 @@
; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
; RUN: opt -passes=lcssa -S %s | FileCheck %s
+; RUN: opt -passes='function(require<memoryssa>,lcssa,verify<memoryssa>)' -S %s | FileCheck %s
; Lifetime markers must point directly to allocas. If an alloca is defined in
; a loop and a marker uses it outside the loop, LCSSA must drop all of the
diff --git a/llvm/unittests/Transforms/Utils/ScalarEvolutionExpanderTest.cpp b/llvm/unittests/Transforms/Utils/ScalarEvolutionExpanderTest.cpp
index f2c0ad7d26a3c..4e4973fbfff05 100644
--- a/llvm/unittests/Transforms/Utils/ScalarEvolutionExpanderTest.cpp
+++ b/llvm/unittests/Transforms/Utils/ScalarEvolutionExpanderTest.cpp
@@ -8,8 +8,11 @@
#include "llvm/Transforms/Utils/ScalarEvolutionExpander.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/Analysis/AliasAnalysis.h"
#include "llvm/Analysis/AssumptionCache.h"
#include "llvm/Analysis/LoopInfo.h"
+#include "llvm/Analysis/MemorySSA.h"
+#include "llvm/Analysis/MemorySSAUpdater.h"
#include "llvm/Analysis/ScalarEvolutionExpressions.h"
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/AsmParser/Parser.h"
@@ -270,6 +273,83 @@ TEST_F(ScalarEvolutionExpanderTest, SCEVExpanderIsSafeToExpandAt) {
EXPECT_TRUE(LI->getLoopFor(L)->isLCSSAForm(*DT));
}
+TEST_F(ScalarEvolutionExpanderTest,
+ PreserveMemorySSAWhenDroppingLifetimeMarkers) {
+ SMDiagnostic Err;
+ std::unique_ptr<Module> Mod = parseAssemblyString(R"(
+ define ptr @f(i1 %again, ptr %out) {
+ entry:
+ br label %loop
+ loop:
+ %allocation = alloca i8
+ call void @llvm.lifetime.start.p0(ptr %allocation)
+ store i8 1, ptr %out
+ br i1 %again, label %loop, label %exit
+ exit:
+ call void @llvm.lifetime.end.p0(ptr %allocation)
+ %loaded = load i8, ptr %out
+ ret ptr null
+ }
+ declare void @llvm.lifetime.start.p0(ptr)
+ declare void @llvm.lifetime.end.p0(ptr)
+ )",
+ Err, Context);
+ ASSERT_NE(Mod, nullptr);
+ ASSERT_FALSE(verifyModule(*Mod, &errs()));
+
+ Function *F = Mod->getFunction("f");
+ ASSERT_NE(F, nullptr);
+ ScalarEvolution SE = buildSE(*F);
+ Instruction &Allocation = GetInstByName(*F, "allocation");
+ Instruction &Load = GetInstByName(*F, "loaded");
+ BasicBlock *LoopBB = Allocation.getParent();
+ BasicBlock *ExitBB = Load.getParent();
+ Loop *L = LI->getLoopFor(LoopBB);
+ ASSERT_NE(L, nullptr);
+ EXPECT_FALSE(L->isLCSSAForm(*DT));
+
+ AAResults AA(TLI);
+ MemorySSA MSSA(*F, &AA, DT.get());
+ MemorySSAUpdater MSSAU(&MSSA);
+ auto *StartAccess = MSSA.getMemoryAccess(Allocation.getNextNode());
+ auto *EndAccess = MSSA.getMemoryAccess(&ExitBB->front());
+ auto *StoreAccess =
+ MSSA.getMemoryAccess(Allocation.getNextNode()->getNextNode());
+ auto *LoadAccess = MSSA.getMemoryAccess(&Load);
+ ASSERT_NE(StartAccess, nullptr);
+ ASSERT_NE(EndAccess, nullptr);
+ ASSERT_NE(StoreAccess, nullptr);
+ ASSERT_NE(LoadAccess, nullptr);
+ EXPECT_EQ(StoreAccess->getDefiningAccess(), StartAccess);
+ EXPECT_EQ(LoadAccess->getDefiningAccess(), EndAccess);
+ MSSA.verifyMemorySSA(MemorySSA::VerificationLevel::Full);
+
+ // Expanding this pointer outside its loop requires an LCSSA PHI. Lifetime
+ // markers cannot use that PHI, so both markers must be removed together with
+ // their MemorySSA accesses, reconnecting the surviving store and load.
+ SCEVExpander Exp(SE, "expander", /*PreserveLCSSA=*/true, &MSSAU);
+ auto *Ret = cast<ReturnInst>(ExitBB->getTerminator());
+ Value *Expanded =
+ Exp.expandCodeFor(SE.getSCEV(&Allocation), Allocation.getType(), Ret);
+ auto *Phi = dyn_cast<PHINode>(Expanded);
+ ASSERT_NE(Phi, nullptr);
+ EXPECT_EQ(Phi->getParent(), ExitBB);
+ ASSERT_EQ(Phi->getNumIncomingValues(), 1u);
+ EXPECT_EQ(Phi->getIncomingValue(0), &Allocation);
+ EXPECT_EQ(Phi->getIncomingBlock(0), LoopBB);
+ Ret->setOperand(0, Phi);
+
+ for (Instruction &I : instructions(*F))
+ EXPECT_FALSE(I.isLifetimeStartOrEnd());
+ EXPECT_EQ(StoreAccess->getDefiningAccess(), MSSA.getMemoryAccess(LoopBB));
+ EXPECT_EQ(LoadAccess->getDefiningAccess(), StoreAccess);
+ ASSERT_NE(MSSA.getBlockAccesses(ExitBB), nullptr);
+ EXPECT_EQ(MSSA.getBlockAccesses(ExitBB)->size(), 1u);
+ EXPECT_TRUE(L->isLCSSAForm(*DT));
+ EXPECT_FALSE(verifyFunction(*F, &errs()));
+ MSSA.verifyMemorySSA(MemorySSA::VerificationLevel::Full);
+}
+
// Check that SCEV expander does not use the nuw instruction
// for expansion.
TEST_F(ScalarEvolutionExpanderTest, SCEVExpanderNUW) {
More information about the llvm-commits
mailing list