[llvm] [AA/MSSA/LICM/Sink] Fix correctness and optimize better around atomics (PR #210568)
Keno Fischer via llvm-commits
llvm-commits at lists.llvm.org
Sat Jul 18 20:29:08 PDT 2026
https://github.com/Keno created https://github.com/llvm/llvm-project/pull/210568
AI Disclosure: The code in this PR was generated by AI (Fable 5), as are the individual commit messages. This PR description is human written.
The primary motivation for this PR from my side is to attempt to allow LICM'ing unrelated loads out of loops that contain `store release` atomic operations. This pattern arises for us (Julia) downstream because we are about to introduce an extra flag read to all global variable writes (https://github.com/JuliaLang/julia/pull/62401). This flag read does not alias with the global write (this fact is available from TBAA) and does not synchronize with it either (although it does synchronize with GC safepoints, which are modeled as global memory clobbers and thus inhibit the optimizations in this PR). Concretely, we want to be able to hoist the load in IR like this out of the loop:
```
loop:
%val = load i32, ptr @flag, align 4 ; <- want to hoist this
store atomic i32 1, ptr @data release, align 4 ; <- past this
[... rest of loop etc]
```
This is legal because a store release operation does not say anything about instructions that are program-order-after the `store release`. Note that the opposite direction does not hold. You cannot sink a load past a `store release` because that may introduce a data race where there was previously none (ask your favorite AI model for a herd7 example).
In LLVM, these restrictions are generally expressed by marking alias information conservative around atomic operations and since alias information does not express this asymmetry, this transformation is currently conservatively disallowed. However, we can run a more precise analysis using MemorySSA. In particular, this PR introduces `getClobberingMemoryAccessForHoist` that works like `getClobberingMemoryAccess` but is only legal for hoisting operations. We then use this in LICM to perform this hoist (although we still run the symmetric one first, since it is cached and thus faster).
In the process of validating these changes, we also noticed a related pre-existing correctness issue where `Sink` would sink `readonly` calls past `store release`. For the same reasons described above, this is illegal. Example:
```llvm
%v = call i32 @read_data(ptr @shared_data) ; readonly, argmem only
store atomic i32 1, ptr @done_flag release
br i1 %c, label %use, label %skip ; %v sunk into %use
```
Fortunately, we can adjust our original LICM patch to at least allow the hoisting of readonly calls out of loops, so I'm submitting both changes together (albeit as separate commits) to avoid regressing performance for users relying on that hoist.
>From ef52566d631041e60b2fd20851075e0088011b38 Mon Sep 17 00:00:00 2001
From: Keno Fischer <Keno at users.noreply.github.com>
Date: Sun, 19 Jul 2026 02:28:31 +0000
Subject: [PATCH 1/2] [AA] Respect atomic ordering in
getModRefInfo(Instruction, CallBase)
AAResults::getModRefInfo(const Instruction *, const CallBase *) only asked
whether the call accesses the instruction's own memory location. For atomic
operations stronger than monotonic that is not sufficient: their
synchronization effects order memory beyond their own location, and the
location-based getModRefInfo overloads already account for this (see
getSyncEffects). The instruction-vs-call overload predates that design and
was never updated.
As a consequence, the Sink pass would sink a read-only call below a release
store (or seq_cst store, or release RMW) that publishes unrelated memory:
```llvm
%v = call i32 @read_data(ptr @shared_data) ; readonly, argmem only
store atomic i32 1, ptr @done_flag release
br i1 %c, label %use, label %skip ; %v sunk into %use
```
With a second thread executing
```c
if (atomic_load_explicit(&flag, memory_order_acquire) == 1) data = 42;
```
the original program is race-free (the call's read happens-before the
release store, which synchronizes-with the acquire, which happens-before
the store to data), while the transformed program has a data race. The
equivalent load-sinking case is already handled correctly because
isSafeToMove queries the location-based store overload.
Fix the overload to conservatively return ModRef when the non-call
instruction is an atomic operation stronger than monotonic and the call may
access memory. Monotonic and unordered atomics impose no cross-location
ordering and keep the precise data-only answer.
Note: LICM's read-only call hoisting reaches this query through MemorySSA's
instructionClobbersQuery and previously hoisted such calls past release and
seq_cst stores only because ordering was ignored here. That hoisting is
legal in the release case (moving a later read earlier), and a follow-up
LICM patch reinstates it with an explicit direction-aware check; this patch
prioritizes fixing the miscompile. No in-tree test relied on the hoisting.
---
llvm/lib/Analysis/AliasAnalysis.cpp | 24 +++++
.../Sink/call-past-release-store.ll | 89 +++++++++++++++++++
2 files changed, 113 insertions(+)
create mode 100644 llvm/test/Transforms/Sink/call-past-release-store.ll
diff --git a/llvm/lib/Analysis/AliasAnalysis.cpp b/llvm/lib/Analysis/AliasAnalysis.cpp
index c7984c35e87ae..7837625f9d90a 100644
--- a/llvm/lib/Analysis/AliasAnalysis.cpp
+++ b/llvm/lib/Analysis/AliasAnalysis.cpp
@@ -201,6 +201,21 @@ ModRefInfo AAResults::getModRefInfo(const Instruction *I,
return getModRefInfo(I, Call2, AAQIP);
}
+/// Return true if \p I is an atomic operation whose ordering is stronger
+/// than monotonic, i.e. one that has synchronization effects on memory
+/// beyond its own location.
+static bool hasOrderingStrongerThanMonotonic(const Instruction *I) {
+ if (auto *LI = dyn_cast<LoadInst>(I))
+ return isStrongerThanMonotonic(LI->getOrdering());
+ if (auto *SI = dyn_cast<StoreInst>(I))
+ return isStrongerThanMonotonic(SI->getOrdering());
+ if (auto *RMW = dyn_cast<AtomicRMWInst>(I))
+ return isStrongerThanMonotonic(RMW->getOrdering());
+ if (auto *CX = dyn_cast<AtomicCmpXchgInst>(I))
+ return isStrongerThanMonotonic(CX->getMergedOrdering());
+ return false;
+}
+
ModRefInfo AAResults::getModRefInfo(const Instruction *I, const CallBase *Call2,
AAQueryInfo &AAQI) {
// We may have two calls.
@@ -211,6 +226,15 @@ ModRefInfo AAResults::getModRefInfo(const Instruction *I, const CallBase *Call2,
// If this is a fence, just return ModRef.
if (I->isFenceLike())
return ModRefInfo::ModRef;
+ // An atomic operation stronger than monotonic also synchronizes with other
+ // threads: like the location-based overloads (see getSyncEffects),
+ // conservatively treat it as ordering any memory the call may access, even
+ // memory disjoint from the atomic's own location. Without this, a client
+ // could e.g. sink a read-only call below a release store that publishes
+ // unrelated memory, breaking the publication guarantee.
+ if (hasOrderingStrongerThanMonotonic(I) &&
+ !getMemoryEffects(Call2, AAQI).doesNotAccessMemory())
+ return ModRefInfo::ModRef;
// Otherwise, check if the call modifies or references the
// location this memory access defines. The best we can say
// is that if the call references what this instruction
diff --git a/llvm/test/Transforms/Sink/call-past-release-store.ll b/llvm/test/Transforms/Sink/call-past-release-store.ll
new file mode 100644
index 0000000000000..76b7387d44446
--- /dev/null
+++ b/llvm/test/Transforms/Sink/call-past-release-store.ll
@@ -0,0 +1,89 @@
+; RUN: opt < %s -passes=sink -S | FileCheck %s
+;
+; A read-only call must not be sunk below an atomic store or RMW with release
+; (or stronger) ordering: moving a program-order-earlier read below a release
+; operation breaks the publication guarantee. Given a second thread executing
+;
+; if (atomic_load_explicit(&flag, memory_order_acquire) == 1)
+; data = 42;
+;
+; the pre-transform functions below are race-free: the call's read of @data
+; happens-before the release operation on @flag, which synchronizes-with the
+; acquire load, which happens-before the second thread's store to @data.
+; Sinking the call below the release operation would leave the read unordered
+; with that store, introducing a data race.
+;
+; Sinking past a monotonic store remains legal: monotonic ordering imposes no
+; constraint on other locations.
+
+ at flag = global i32 0
+ at data = global i32 0
+
+declare i32 @read_data(ptr) nounwind willreturn memory(argmem: read)
+
+define i32 @no_sink_call_past_release_store(i1 %c) {
+; CHECK-LABEL: @no_sink_call_past_release_store(
+; CHECK: [[V:%.*]] = call i32 @read_data(ptr @data)
+; CHECK-NEXT: store atomic i32 1, ptr @flag release
+entry:
+ %v = call i32 @read_data(ptr @data)
+ store atomic i32 1, ptr @flag release, align 4
+ br i1 %c, label %use, label %skip
+
+use:
+ ret i32 %v
+
+skip:
+ ret i32 0
+}
+
+define i32 @no_sink_call_past_seq_cst_store(i1 %c) {
+; CHECK-LABEL: @no_sink_call_past_seq_cst_store(
+; CHECK: [[V:%.*]] = call i32 @read_data(ptr @data)
+; CHECK-NEXT: store atomic i32 1, ptr @flag seq_cst
+entry:
+ %v = call i32 @read_data(ptr @data)
+ store atomic i32 1, ptr @flag seq_cst, align 4
+ br i1 %c, label %use, label %skip
+
+use:
+ ret i32 %v
+
+skip:
+ ret i32 0
+}
+
+define i32 @no_sink_call_past_release_rmw(i1 %c) {
+; CHECK-LABEL: @no_sink_call_past_release_rmw(
+; CHECK: [[V:%.*]] = call i32 @read_data(ptr @data)
+; CHECK-NEXT: {{%.*}} = atomicrmw add ptr @flag, i32 1 release
+entry:
+ %v = call i32 @read_data(ptr @data)
+ %old = atomicrmw add ptr @flag, i32 1 release, align 4
+ br i1 %c, label %use, label %skip
+
+use:
+ ret i32 %v
+
+skip:
+ ret i32 0
+}
+
+; Monotonic imposes no cross-location ordering; the call may be sunk.
+define i32 @sink_call_past_monotonic_store(i1 %c) {
+; CHECK-LABEL: @sink_call_past_monotonic_store(
+; CHECK: entry:
+; CHECK-NEXT: store atomic i32 1, ptr @flag monotonic
+; CHECK: use:
+; CHECK-NEXT: [[V:%.*]] = call i32 @read_data(ptr @data)
+entry:
+ %v = call i32 @read_data(ptr @data)
+ store atomic i32 1, ptr @flag monotonic, align 4
+ br i1 %c, label %use, label %skip
+
+use:
+ ret i32 %v
+
+skip:
+ ret i32 0
+}
>From 40e5a1991524d6756b567ab06797bdb28bb406e0 Mon Sep 17 00:00:00 2001
From: Keno Fischer <Keno at users.noreply.github.com>
Date: Sun, 19 Jul 2026 02:54:00 +0000
Subject: [PATCH 2/2] [MemorySSA][LICM] Add hoist-only clobber walk that skips
non-aliasing release stores
A store with release (or weaker) ordering constrains program-order-earlier
accesses only; AA nevertheless reports it as clobbering every escaped
location (see getSyncEffects) so that direction-agnostic clients do not
illegally sink accesses below it or delete accesses above it. For clients
that exclusively move the queried access to an earlier program point, that
ordering component does not apply, but the information is buried inside the
clobber walk: with the release store on the back edge, the walker returns
the loop header MemoryPhi, and the caller cannot tell whether the verdict
stems from a real data clobber or from ordering alone.
Add MemorySSAWalker::getClobberingMemoryAccessForHoist: a clobber walk
that treats a non-volatile release store def as non-clobbering when it
provably has no data effect on the queried access (for a load, the
locations cannot alias; for a call, the call does not access the store's
location). The flag lives on UpwardsMemoryQuery and is honored in
instructionClobbersQuery, so both the def-chain walk and phi optimization
see it uniformly. Results computed this way are only valid for hoisting,
so the new entry point bypasses the isOptimized cache in both directions;
default-flag queries are unchanged.
Use it in LICM's hoist path: when the cached walker query reports an
in-loop clobber for an unordered load or a read-only call, retry with the
hoist walk. This allows hoisting loop-invariant loads and read-only calls
out of loops that publish unrelated data via release stores (the
publication idiom), and reinstates the read-only call hoisting that the
preceding getModRefInfo(Instruction, CallBase) fix conservatively removed.
The sink path is untouched and continues to refuse moving a load below any
in-loop MemoryDef. seq_cst stores, volatile atomic stores, ordered loads,
atomicrmw, cmpxchg and fences all remain hoisting barriers.
Moving a later read above a release store is explicitly permitted by the
memory model (Atomics.rst, Release, notes for optimizers); MemoryDependence
already implements the equivalent direction-aware reasoning for GVN
(D119844).
---
llvm/include/llvm/Analysis/MemorySSA.h | 18 +
llvm/lib/Analysis/MemorySSA.cpp | 90 ++++-
llvm/lib/Transforms/Scalar/LICM.cpp | 36 +-
.../LICM/hoist-call-past-release-store.ll | 82 +++++
.../LICM/hoist-load-past-release-store.ll | 332 ++++++++++++++++++
5 files changed, 551 insertions(+), 7 deletions(-)
create mode 100644 llvm/test/Transforms/LICM/hoist-call-past-release-store.ll
create mode 100644 llvm/test/Transforms/LICM/hoist-load-past-release-store.ll
diff --git a/llvm/include/llvm/Analysis/MemorySSA.h b/llvm/include/llvm/Analysis/MemorySSA.h
index b95a8d0a80e9f..e59f0840a41fe 100644
--- a/llvm/include/llvm/Analysis/MemorySSA.h
+++ b/llvm/include/llvm/Analysis/MemorySSA.h
@@ -1075,6 +1075,22 @@ class MemorySSAWalker {
return getClobberingMemoryAccess(MA, Loc, BAA);
}
+ /// Does the same thing as getClobberingMemoryAccess(MA, BAA), except that
+ /// atomic store defs with release (or weaker) ordering that provably have
+ /// no data effect on the queried access are not treated as clobbers.
+ ///
+ /// Such stores are normally reported as clobbering every escaped location
+ /// (see getSyncEffects) solely because their ordering forbids moving
+ /// program-order-earlier accesses below them. They impose nothing on
+ /// program-order-later reads, so the result of this query is valid ONLY
+ /// for clients that move the queried access to an *earlier* program point
+ /// (hoisting): it must not be used to justify sinking any access below,
+ /// or deleting any access above, one of the skipped stores.
+ ///
+ /// The result is never cached, and cached results are not consulted.
+ virtual MemoryAccess *getClobberingMemoryAccessForHoist(MemoryUseOrDef *,
+ BatchAAResults &) = 0;
+
/// Given a memory access, invalidate anything this walker knows about
/// that access.
/// This API is used by walkers that store information to perform basic cache
@@ -1101,6 +1117,8 @@ class LLVM_ABI DoNothingMemorySSAWalker final : public MemorySSAWalker {
MemoryAccess *getClobberingMemoryAccess(MemoryAccess *,
const MemoryLocation &,
BatchAAResults &) override;
+ MemoryAccess *getClobberingMemoryAccessForHoist(MemoryUseOrDef *,
+ BatchAAResults &) override;
};
/// Iterator base class used to implement const and non-const iterators
diff --git a/llvm/lib/Analysis/MemorySSA.cpp b/llvm/lib/Analysis/MemorySSA.cpp
index ebe10b9073a00..f24f68a3c5306 100644
--- a/llvm/lib/Analysis/MemorySSA.cpp
+++ b/llvm/lib/Analysis/MemorySSA.cpp
@@ -273,10 +273,32 @@ static bool areLoadsReorderable(const LoadInst *Use,
template <typename AliasAnalysisType>
static bool
instructionClobbersQuery(const MemoryDef *MD, const MemoryLocation &UseLoc,
- const Instruction *UseInst, AliasAnalysisType &AA) {
+ const Instruction *UseInst, AliasAnalysisType &AA,
+ bool SkipNonAliasingReleaseStores = false) {
Instruction *DefInst = MD->getMemoryInst();
assert(DefInst && "Defining instruction not actually an instruction");
+ // A store with release (or weaker) ordering constrains program-order-earlier
+ // accesses only. It is reported as clobbering escaped locations it cannot
+ // alias (see getSyncEffects) solely so that accesses are not illegally
+ // moved below it or deleted above it. A walk on behalf of a client that
+ // only moves the queried access to an *earlier* program point may opt out
+ // of that ordering component for stores that provably have no data effect
+ // on the queried access; see getClobberingMemoryAccessForHoist.
+ if (SkipNonAliasingReleaseStores) {
+ if (auto *SI = dyn_cast<StoreInst>(DefInst);
+ SI && SI->getOrdering() == AtomicOrdering::Release &&
+ !SI->isVolatile()) {
+ // For a call, the store has no data effect if the call does not access
+ // the store's location; for a load, if the locations cannot alias.
+ if (auto *CB = dyn_cast_or_null<CallBase>(UseInst))
+ return isModOrRefSet(AA.getModRefInfo(CB, MemoryLocation::get(SI)));
+ if (UseLoc.Ptr &&
+ AA.alias(MemoryLocation::get(SI), UseLoc) == AliasResult::NoAlias)
+ return false;
+ }
+ }
+
if (const IntrinsicInst *II = dyn_cast<IntrinsicInst>(DefInst)) {
// These intrinsics will show up as affecting memory, but they are just
// markers, mostly.
@@ -348,6 +370,10 @@ struct UpwardsMemoryQuery {
// The MemoryAccess we actually got called with, used to test local domination
const MemoryAccess *OriginalAccess = nullptr;
bool SkipSelfAccess = false;
+ // Treat release-or-weaker store defs with no data effect on the queried
+ // access as non-clobbering. Only valid for hoisting; results computed with
+ // this set must never be cached. See getClobberingMemoryAccessForHoist.
+ bool SkipNonAliasingReleaseStores = false;
UpwardsMemoryQuery() = default;
@@ -423,7 +449,8 @@ checkClobberSanity(MemoryAccess *Start, MemoryAccess *ClobberAt,
FoundClobber = FoundClobber || MSSA.isLiveOnEntryDef(MD);
if (!FoundClobber) {
BatchAACrossIterationScope _(AA, MAP.MayBeCrossIteration);
- if (instructionClobbersQuery(MD, MAP.Loc, Query.Inst, AA))
+ if (instructionClobbersQuery(MD, MAP.Loc, Query.Inst, AA,
+ Query.SkipNonAliasingReleaseStores))
FoundClobber = true;
}
}
@@ -439,7 +466,8 @@ checkClobberSanity(MemoryAccess *Start, MemoryAccess *ClobberAt,
continue;
BatchAACrossIterationScope _(AA, MAP.MayBeCrossIteration);
- assert(!instructionClobbersQuery(MD, MAP.Loc, Query.Inst, AA) &&
+ assert(!instructionClobbersQuery(MD, MAP.Loc, Query.Inst, AA,
+ Query.SkipNonAliasingReleaseStores) &&
"Found clobber before reaching ClobberAt!");
continue;
}
@@ -575,7 +603,8 @@ class ClobberWalker {
return {Current, true};
BatchAACrossIterationScope _(*AA, Desc.MayBeCrossIteration);
- if (instructionClobbersQuery(MD, Desc.Loc, Query->Inst, *AA))
+ if (instructionClobbersQuery(MD, Desc.Loc, Query->Inst, *AA,
+ Query->SkipNonAliasingReleaseStores))
return {MD, true};
}
}
@@ -1007,6 +1036,12 @@ class MemorySSA::ClobberWalkerBase {
MemoryAccess *getClobberingMemoryAccessBase(MemoryAccess *, BatchAAResults &,
unsigned &, bool,
bool UseInvariantGroup = true);
+ // Uncached walk that skips non-aliasing release-or-weaker store defs; only
+ // valid for hoisting the queried access. See
+ // MemorySSAWalker::getClobberingMemoryAccessForHoist.
+ MemoryAccess *getClobberingMemoryAccessForHoistBase(MemoryUseOrDef *,
+ BatchAAResults &,
+ unsigned &);
};
/// A MemorySSAWalker that does AA walks to disambiguate accesses. It no
@@ -1049,6 +1084,13 @@ class MemorySSA::CachingWalker final : public MemorySSAWalker {
return getClobberingMemoryAccess(MA, Loc, BAA, UpwardWalkLimit);
}
+ MemoryAccess *getClobberingMemoryAccessForHoist(MemoryUseOrDef *MA,
+ BatchAAResults &BAA) override {
+ unsigned UpwardWalkLimit = MaxCheckLimit;
+ return Walker->getClobberingMemoryAccessForHoistBase(MA, BAA,
+ UpwardWalkLimit);
+ }
+
void invalidateInfo(MemoryAccess *MA) override {
if (auto *MUD = dyn_cast<MemoryUseOrDef>(MA))
MUD->resetOptimized();
@@ -1087,6 +1129,13 @@ class MemorySSA::SkipSelfWalker final : public MemorySSAWalker {
return getClobberingMemoryAccess(MA, Loc, BAA, UpwardWalkLimit);
}
+ MemoryAccess *getClobberingMemoryAccessForHoist(MemoryUseOrDef *MA,
+ BatchAAResults &BAA) override {
+ unsigned UpwardWalkLimit = MaxCheckLimit;
+ return Walker->getClobberingMemoryAccessForHoistBase(MA, BAA,
+ UpwardWalkLimit);
+ }
+
void invalidateInfo(MemoryAccess *MA) override {
if (auto *MUD = dyn_cast<MemoryUseOrDef>(MA))
MUD->resetOptimized();
@@ -2612,6 +2661,34 @@ MemoryAccess *MemorySSA::ClobberWalkerBase::getClobberingMemoryAccessBase(
return Result;
}
+MemoryAccess *
+MemorySSA::ClobberWalkerBase::getClobberingMemoryAccessForHoistBase(
+ MemoryUseOrDef *StartingAccess, BatchAAResults &BAA,
+ unsigned &UpwardWalkLimit) {
+ const Instruction *I = StartingAccess->getMemoryInst();
+
+ // We can't sanely do anything with a fence; conservatively return the
+ // access itself, like the location-based walk.
+ if (!isa<CallBase>(I) && I->isFenceLike())
+ return StartingAccess;
+
+ if (isUseTriviallyOptimizableToLiveOnEntry(BAA, I))
+ return MSSA->getLiveOnEntryDef();
+
+ MemoryAccess *DefiningAccess = StartingAccess->getDefiningAccess();
+ if (MSSA->isLiveOnEntryDef(DefiningAccess))
+ return DefiningAccess;
+
+ UpwardsMemoryQuery Q(I, StartingAccess);
+ Q.SkipNonAliasingReleaseStores = true;
+
+ // Deliberately bypass the isOptimized()/setOptimized() cache in both
+ // directions: results computed while skipping release stores are only
+ // valid for hoisting the queried access and must never be visible to any
+ // other client.
+ return Walker.findClobber(BAA, DefiningAccess, Q, UpwardWalkLimit);
+}
+
MemoryAccess *
DoNothingMemorySSAWalker::getClobberingMemoryAccess(MemoryAccess *MA,
BatchAAResults &) {
@@ -2620,6 +2697,11 @@ DoNothingMemorySSAWalker::getClobberingMemoryAccess(MemoryAccess *MA,
return MA;
}
+MemoryAccess *DoNothingMemorySSAWalker::getClobberingMemoryAccessForHoist(
+ MemoryUseOrDef *MA, BatchAAResults &) {
+ return MA->getDefiningAccess();
+}
+
MemoryAccess *DoNothingMemorySSAWalker::getClobberingMemoryAccess(
MemoryAccess *StartingAccess, const MemoryLocation &, BatchAAResults &) {
if (auto *Use = dyn_cast<MemoryUseOrDef>(StartingAccess))
diff --git a/llvm/lib/Transforms/Scalar/LICM.cpp b/llvm/lib/Transforms/Scalar/LICM.cpp
index c7e29b70662d8..7501b28363af9 100644
--- a/llvm/lib/Transforms/Scalar/LICM.cpp
+++ b/llvm/lib/Transforms/Scalar/LICM.cpp
@@ -2373,10 +2373,40 @@ static bool pointerInvalidatedByLoop(MemorySSA *MSSA, MemoryUse *MU,
// if the memory loaded is the phi node
BatchAAResults BAA(MSSA->getAA());
+ // Fast path: this query is answered from MemorySSA's cache (uses are
+ // optimized at construction time), unlike the uncached
+ // getClobberingMemoryAccessForHoist walk below, which we therefore only
+ // run if this conservative verdict blocks hoisting.
MemoryAccess *Source = getClobberingMemoryAccess(*MSSA, BAA, Flags, MU);
- return !MSSA->isLiveOnEntryDef(Source) &&
- CurLoop->contains(Source->getBlock()) &&
- !(InvariantGroup && Source->getBlock() == CurLoop->getHeader() && isa<MemoryPhi>(Source));
+ bool Invalidated =
+ !MSSA->isLiveOnEntryDef(Source) &&
+ CurLoop->contains(Source->getBlock()) &&
+ !(InvariantGroup && Source->getBlock() == CurLoop->getHeader() &&
+ isa<MemoryPhi>(Source));
+
+ // The walker treats release stores as clobbers of any escaped location
+ // (see getSyncEffects) because their ordering forbids moving
+ // program-order-earlier accesses below them; that blocks hoisting loads
+ // and read-only calls out of loops that publish unrelated data. Hoisting
+ // only moves the access to an earlier program point, which a
+ // release-or-weaker store does not constrain, so retry the walk with
+ // such stores skipped when they provably have no data effect on the
+ // access. This must only be done when hoisting: the sink path below
+ // stays conservative, as sinking below a release store is illegal.
+ if (Invalidated && !Flags.tooManyClobberingCalls()) {
+ auto *LI = dyn_cast<LoadInst>(&I);
+ if ((LI && LI->isUnordered()) || (!LI && isa<CallBase>(&I))) {
+ Source = MSSA->getWalker()->getClobberingMemoryAccessForHoist(MU, BAA);
+ Flags.incrementClobberingCalls();
+ Invalidated =
+ !MSSA->isLiveOnEntryDef(Source) &&
+ CurLoop->contains(Source->getBlock()) &&
+ !(InvariantGroup && Source->getBlock() == CurLoop->getHeader() &&
+ isa<MemoryPhi>(Source));
+ }
+ }
+
+ return Invalidated;
}
// For sinking, we'd need to check all Defs below this use. The getClobbering
diff --git a/llvm/test/Transforms/LICM/hoist-call-past-release-store.ll b/llvm/test/Transforms/LICM/hoist-call-past-release-store.ll
new file mode 100644
index 0000000000000..2a296ad70c80e
--- /dev/null
+++ b/llvm/test/Transforms/LICM/hoist-call-past-release-store.ll
@@ -0,0 +1,82 @@
+; RUN: opt -passes='loop-mssa(licm)' -S < %s | FileCheck %s
+;
+; A read-only call, like a load, may be hoisted above a release store to
+; memory it does not access: the release ordering constrains only
+; program-order-earlier accesses. Keep seq_cst stores and release RMWs as
+; conservative barriers.
+
+ at flag = global i32 0
+ at data = global i32 0
+
+declare i32 @read_data(ptr) nounwind willreturn memory(argmem: read)
+
+define i32 @hoist_call_past_release_store(i32 %n) {
+; CHECK-LABEL: @hoist_call_past_release_store(
+; CHECK: entry:
+; CHECK-NEXT: [[V:%.*]] = call i32 @read_data(ptr @data)
+; CHECK-NEXT: br label %loop
+; CHECK: loop:
+; CHECK-NOT: call i32 @read_data
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+ %sum = phi i32 [ 0, %entry ], [ %sum.next, %loop ]
+ %v = call i32 @read_data(ptr @data)
+ %sum.next = add i32 %sum, %v
+ store atomic i32 1, ptr @flag release, align 4
+ %i.next = add nuw nsw i32 %i, 1
+ %cmp = icmp slt i32 %i.next, %n
+ br i1 %cmp, label %loop, label %exit
+
+exit:
+ ret i32 %sum.next
+}
+
+; Negative: seq_cst stores stay conservative.
+define i32 @no_hoist_call_past_seq_cst_store(i32 %n) {
+; CHECK-LABEL: @no_hoist_call_past_seq_cst_store(
+; CHECK: loop:
+; CHECK: call i32 @read_data(ptr @data)
+; CHECK-NEXT: {{%.*}} = add
+; CHECK-NEXT: store atomic i32 1, ptr @flag seq_cst
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+ %sum = phi i32 [ 0, %entry ], [ %sum.next, %loop ]
+ %v = call i32 @read_data(ptr @data)
+ %sum.next = add i32 %sum, %v
+ store atomic i32 1, ptr @flag seq_cst, align 4
+ %i.next = add nuw nsw i32 %i, 1
+ %cmp = icmp slt i32 %i.next, %n
+ br i1 %cmp, label %loop, label %exit
+
+exit:
+ ret i32 %sum.next
+}
+
+; Negative: the call may read the location the release store writes; that is
+; a real data dependence, not just ordering.
+define i32 @no_hoist_call_reads_flag(i32 %n) {
+; CHECK-LABEL: @no_hoist_call_reads_flag(
+; CHECK: loop:
+; CHECK: call i32 @read_data(ptr @flag)
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+ %sum = phi i32 [ 0, %entry ], [ %sum.next, %loop ]
+ %v = call i32 @read_data(ptr @flag)
+ %sum.next = add i32 %sum, %v
+ store atomic i32 1, ptr @flag release, align 4
+ %i.next = add nuw nsw i32 %i, 1
+ %cmp = icmp slt i32 %i.next, %n
+ br i1 %cmp, label %loop, label %exit
+
+exit:
+ ret i32 %sum.next
+}
diff --git a/llvm/test/Transforms/LICM/hoist-load-past-release-store.ll b/llvm/test/Transforms/LICM/hoist-load-past-release-store.ll
new file mode 100644
index 0000000000000..4bd6f67fc4844
--- /dev/null
+++ b/llvm/test/Transforms/LICM/hoist-load-past-release-store.ll
@@ -0,0 +1,332 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -passes='loop-mssa(licm)' -S < %s | FileCheck %s
+
+; A store with release (or weaker) ordering imposes no constraint on
+; program-order-later reads of unrelated memory: moving a load from after a
+; release store to before it is explicitly permitted by the memory model.
+; Check that LICM hoists loop-invariant loads out of loops whose body contains
+; a release store to provably non-aliasing memory.
+
+ at flag = global i32 0
+ at data = global i32 0
+
+; The load of @data is loop invariant; the release store to @flag must not
+; block hoisting it into the preheader.
+define i32 @hoist_past_release_store(i32 %n) {
+; CHECK-LABEL: @hoist_past_release_store(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: [[VAL:%.*]] = load i32, ptr @data, align 4
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[I_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[SUM:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[SUM_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[SUM_NEXT]] = add i32 [[SUM]], [[VAL]]
+; CHECK-NEXT: store atomic i32 1, ptr @flag release, align 4
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i32 [[I]], 1
+; CHECK-NEXT: [[CMP:%.*]] = icmp slt i32 [[I_NEXT]], [[N:%.*]]
+; CHECK-NEXT: br i1 [[CMP]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK: exit:
+; CHECK-NEXT: [[SUM_NEXT_LCSSA:%.*]] = phi i32 [ [[SUM_NEXT]], [[LOOP]] ]
+; CHECK-NEXT: ret i32 [[SUM_NEXT_LCSSA]]
+;
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+ %sum = phi i32 [ 0, %entry ], [ %sum.next, %loop ]
+ %val = load i32, ptr @data, align 4
+ %sum.next = add i32 %sum, %val
+ store atomic i32 1, ptr @flag release, align 4
+ %i.next = add nuw nsw i32 %i, 1
+ %cmp = icmp slt i32 %i.next, %n
+ br i1 %cmp, label %loop, label %exit
+
+exit:
+ ret i32 %sum.next
+}
+
+; Same for a monotonic store.
+define i32 @hoist_past_monotonic_store(i32 %n) {
+; CHECK-LABEL: @hoist_past_monotonic_store(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: [[VAL:%.*]] = load i32, ptr @data, align 4
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[I_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[SUM:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[SUM_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[SUM_NEXT]] = add i32 [[SUM]], [[VAL]]
+; CHECK-NEXT: store atomic i32 1, ptr @flag monotonic, align 4
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i32 [[I]], 1
+; CHECK-NEXT: [[CMP:%.*]] = icmp slt i32 [[I_NEXT]], [[N:%.*]]
+; CHECK-NEXT: br i1 [[CMP]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK: exit:
+; CHECK-NEXT: [[SUM_NEXT_LCSSA:%.*]] = phi i32 [ [[SUM_NEXT]], [[LOOP]] ]
+; CHECK-NEXT: ret i32 [[SUM_NEXT_LCSSA]]
+;
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+ %sum = phi i32 [ 0, %entry ], [ %sum.next, %loop ]
+ %val = load i32, ptr @data, align 4
+ %sum.next = add i32 %sum, %val
+ store atomic i32 1, ptr @flag monotonic, align 4
+ %i.next = add nuw nsw i32 %i, 1
+ %cmp = icmp slt i32 %i.next, %n
+ br i1 %cmp, label %loop, label %exit
+
+exit:
+ ret i32 %sum.next
+}
+
+; Multiple release stores are all skipped.
+define i32 @hoist_past_two_release_stores(i32 %n) {
+; CHECK-LABEL: @hoist_past_two_release_stores(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: [[VAL:%.*]] = load i32, ptr @data, align 4
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[I_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[SUM:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[SUM_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: store atomic i32 0, ptr @flag release, align 4
+; CHECK-NEXT: [[SUM_NEXT]] = add i32 [[SUM]], [[VAL]]
+; CHECK-NEXT: store atomic i32 1, ptr @flag release, align 4
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i32 [[I]], 1
+; CHECK-NEXT: [[CMP:%.*]] = icmp slt i32 [[I_NEXT]], [[N:%.*]]
+; CHECK-NEXT: br i1 [[CMP]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK: exit:
+; CHECK-NEXT: [[SUM_NEXT_LCSSA:%.*]] = phi i32 [ [[SUM_NEXT]], [[LOOP]] ]
+; CHECK-NEXT: ret i32 [[SUM_NEXT_LCSSA]]
+;
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+ %sum = phi i32 [ 0, %entry ], [ %sum.next, %loop ]
+ store atomic i32 0, ptr @flag release, align 4
+ %val = load i32, ptr @data, align 4
+ %sum.next = add i32 %sum, %val
+ store atomic i32 1, ptr @flag release, align 4
+ %i.next = add nuw nsw i32 %i, 1
+ %cmp = icmp slt i32 %i.next, %n
+ br i1 %cmp, label %loop, label %exit
+
+exit:
+ ret i32 %sum.next
+}
+
+; Negative: a seq_cst store also participates in the total order of seq_cst
+; operations; stay conservative.
+define i32 @no_hoist_past_seq_cst_store(i32 %n) {
+; CHECK-LABEL: @no_hoist_past_seq_cst_store(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[I_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[SUM:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[SUM_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[VAL:%.*]] = load i32, ptr @data, align 4
+; CHECK-NEXT: [[SUM_NEXT]] = add i32 [[SUM]], [[VAL]]
+; CHECK-NEXT: store atomic i32 1, ptr @flag seq_cst, align 4
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i32 [[I]], 1
+; CHECK-NEXT: [[CMP:%.*]] = icmp slt i32 [[I_NEXT]], [[N:%.*]]
+; CHECK-NEXT: br i1 [[CMP]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK: exit:
+; CHECK-NEXT: [[SUM_NEXT_LCSSA:%.*]] = phi i32 [ [[SUM_NEXT]], [[LOOP]] ]
+; CHECK-NEXT: ret i32 [[SUM_NEXT_LCSSA]]
+;
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+ %sum = phi i32 [ 0, %entry ], [ %sum.next, %loop ]
+ %val = load i32, ptr @data, align 4
+ %sum.next = add i32 %sum, %val
+ store atomic i32 1, ptr @flag seq_cst, align 4
+ %i.next = add nuw nsw i32 %i, 1
+ %cmp = icmp slt i32 %i.next, %n
+ br i1 %cmp, label %loop, label %exit
+
+exit:
+ ret i32 %sum.next
+}
+
+; Negative: the release store may alias the load; it is a real data clobber.
+define i32 @no_hoist_may_alias(ptr %p, ptr %q, i32 %n) {
+; CHECK-LABEL: @no_hoist_may_alias(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[I_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[SUM:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[SUM_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[VAL:%.*]] = load i32, ptr [[P:%.*]], align 4
+; CHECK-NEXT: [[SUM_NEXT]] = add i32 [[SUM]], [[VAL]]
+; CHECK-NEXT: store atomic i32 1, ptr [[Q:%.*]] release, align 4
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i32 [[I]], 1
+; CHECK-NEXT: [[CMP:%.*]] = icmp slt i32 [[I_NEXT]], [[N:%.*]]
+; CHECK-NEXT: br i1 [[CMP]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK: exit:
+; CHECK-NEXT: [[SUM_NEXT_LCSSA:%.*]] = phi i32 [ [[SUM_NEXT]], [[LOOP]] ]
+; CHECK-NEXT: ret i32 [[SUM_NEXT_LCSSA]]
+;
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+ %sum = phi i32 [ 0, %entry ], [ %sum.next, %loop ]
+ %val = load i32, ptr %p, align 4
+ %sum.next = add i32 %sum, %val
+ store atomic i32 1, ptr %q release, align 4
+ %i.next = add nuw nsw i32 %i, 1
+ %cmp = icmp slt i32 %i.next, %n
+ br i1 %cmp, label %loop, label %exit
+
+exit:
+ ret i32 %sum.next
+}
+
+; Negative: an aliasing plain store behind the skipped release store is still
+; found by the continued walk.
+define i32 @no_hoist_clobber_behind_release_store(i32 %n) {
+; CHECK-LABEL: @no_hoist_clobber_behind_release_store(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[I_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[SUM:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[SUM_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: store i32 [[I]], ptr @data, align 4
+; CHECK-NEXT: store atomic i32 1, ptr @flag release, align 4
+; CHECK-NEXT: [[VAL:%.*]] = load i32, ptr @data, align 4
+; CHECK-NEXT: [[SUM_NEXT]] = add i32 [[SUM]], [[VAL]]
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i32 [[I]], 1
+; CHECK-NEXT: [[CMP:%.*]] = icmp slt i32 [[I_NEXT]], [[N:%.*]]
+; CHECK-NEXT: br i1 [[CMP]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK: exit:
+; CHECK-NEXT: [[SUM_NEXT_LCSSA:%.*]] = phi i32 [ [[SUM_NEXT]], [[LOOP]] ]
+; CHECK-NEXT: ret i32 [[SUM_NEXT_LCSSA]]
+;
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+ %sum = phi i32 [ 0, %entry ], [ %sum.next, %loop ]
+ store i32 %i, ptr @data, align 4
+ store atomic i32 1, ptr @flag release, align 4
+ %val = load i32, ptr @data, align 4
+ %sum.next = add i32 %sum, %val
+ %i.next = add nuw nsw i32 %i, 1
+ %cmp = icmp slt i32 %i.next, %n
+ br i1 %cmp, label %loop, label %exit
+
+exit:
+ ret i32 %sum.next
+}
+
+; Negative: volatile atomic stores are not skipped.
+define i32 @no_hoist_past_volatile_release_store(i32 %n) {
+; CHECK-LABEL: @no_hoist_past_volatile_release_store(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[I_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[SUM:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[SUM_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[VAL:%.*]] = load i32, ptr @data, align 4
+; CHECK-NEXT: [[SUM_NEXT]] = add i32 [[SUM]], [[VAL]]
+; CHECK-NEXT: store atomic volatile i32 1, ptr @flag release, align 4
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i32 [[I]], 1
+; CHECK-NEXT: [[CMP:%.*]] = icmp slt i32 [[I_NEXT]], [[N:%.*]]
+; CHECK-NEXT: br i1 [[CMP]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK: exit:
+; CHECK-NEXT: [[SUM_NEXT_LCSSA:%.*]] = phi i32 [ [[SUM_NEXT]], [[LOOP]] ]
+; CHECK-NEXT: ret i32 [[SUM_NEXT_LCSSA]]
+;
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+ %sum = phi i32 [ 0, %entry ], [ %sum.next, %loop ]
+ %val = load i32, ptr @data, align 4
+ %sum.next = add i32 %sum, %val
+ store atomic volatile i32 1, ptr @flag release, align 4
+ %i.next = add nuw nsw i32 %i, 1
+ %cmp = icmp slt i32 %i.next, %n
+ br i1 %cmp, label %loop, label %exit
+
+exit:
+ ret i32 %sum.next
+}
+
+; Negative: only plain stores are skipped; a release atomicrmw is left as a
+; conservative barrier.
+define i32 @no_hoist_past_release_rmw(i32 %n) {
+; CHECK-LABEL: @no_hoist_past_release_rmw(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[I_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[SUM:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[SUM_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[VAL:%.*]] = load i32, ptr @data, align 4
+; CHECK-NEXT: [[SUM_NEXT]] = add i32 [[SUM]], [[VAL]]
+; CHECK-NEXT: [[OLD:%.*]] = atomicrmw add ptr @flag, i32 1 release, align 4
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i32 [[I]], 1
+; CHECK-NEXT: [[CMP:%.*]] = icmp slt i32 [[I_NEXT]], [[N:%.*]]
+; CHECK-NEXT: br i1 [[CMP]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK: exit:
+; CHECK-NEXT: [[SUM_NEXT_LCSSA:%.*]] = phi i32 [ [[SUM_NEXT]], [[LOOP]] ]
+; CHECK-NEXT: ret i32 [[SUM_NEXT_LCSSA]]
+;
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+ %sum = phi i32 [ 0, %entry ], [ %sum.next, %loop ]
+ %val = load i32, ptr @data, align 4
+ %sum.next = add i32 %sum, %val
+ %old = atomicrmw add ptr @flag, i32 1 release, align 4
+ %i.next = add nuw nsw i32 %i, 1
+ %cmp = icmp slt i32 %i.next, %n
+ br i1 %cmp, label %loop, label %exit
+
+exit:
+ ret i32 %sum.next
+}
+
+; A load whose only use is outside the loop must not be *sunk* below the
+; release store; it is hoisted to the preheader instead (LICM's sink path
+; refuses to move a load below any MemoryDef in the loop).
+define i32 @no_sink_used_outside(i32 %n) {
+; CHECK-LABEL: @no_sink_used_outside(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: [[VAL:%.*]] = load i32, ptr @data, align 4
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[I_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: store atomic i32 1, ptr @flag release, align 4
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i32 [[I]], 1
+; CHECK-NEXT: [[CMP:%.*]] = icmp slt i32 [[I_NEXT]], [[N:%.*]]
+; CHECK-NEXT: br i1 [[CMP]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK: exit:
+; CHECK-NEXT: [[VAL_LCSSA:%.*]] = phi i32 [ [[VAL]], [[LOOP]] ]
+; CHECK-NEXT: ret i32 [[VAL_LCSSA]]
+;
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+ %val = load i32, ptr @data, align 4
+ store atomic i32 1, ptr @flag release, align 4
+ %i.next = add nuw nsw i32 %i, 1
+ %cmp = icmp slt i32 %i.next, %n
+ br i1 %cmp, label %loop, label %exit
+
+exit:
+ ret i32 %val
+}
More information about the llvm-commits
mailing list