[llvm] [AA/MSSA/LICM/Sink] Fix correctness and optimize better around atomics (PR #210568)
via llvm-commits
llvm-commits at lists.llvm.org
Sat Jul 18 20:29:42 PDT 2026
llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-llvm-transforms
Author: Keno Fischer (Keno)
<details>
<summary>Changes</summary>
AI Disclosure: The code in this PR was generated by AI (Fable 5), as are the individual commit messages. This PR description is human written.
The primary motivation for this PR from my side is to attempt to allow LICM'ing unrelated loads out of loops that contain `store release` atomic operations. This pattern arises for us (Julia) downstream because we are about to introduce an extra flag read to all global variable writes (https://github.com/JuliaLang/julia/pull/62401). This flag read does not alias with the global write (this fact is available from TBAA) and does not synchronize with it either (although it does synchronize with GC safepoints, which are modeled as global memory clobbers and thus inhibit the optimizations in this PR). Concretely, we want to be able to hoist the load in IR like this out of the loop:
```
loop:
%val = load i32, ptr @<!-- -->flag, align 4 ; <- want to hoist this
store atomic i32 1, ptr @<!-- -->data release, align 4 ; <- past this
[... rest of loop etc]
```
This is legal because a store release operation does not say anything about instructions that are program-order-after the `store release`. Note that the opposite direction does not hold. You cannot sink a load past a `store release` because that may introduce a data race where there was previously none (ask your favorite AI model for a herd7 example).
In LLVM, these restrictions are generally expressed by marking alias information conservative around atomic operations and since alias information does not express this asymmetry, this transformation is currently conservatively disallowed. However, we can run a more precise analysis using MemorySSA. In particular, this PR introduces `getClobberingMemoryAccessForHoist` that works like `getClobberingMemoryAccess` but is only legal for hoisting operations. We then use this in LICM to perform this hoist (although we still run the symmetric one first, since it is cached and thus faster).
In the process of validating these changes, we also noticed a related pre-existing correctness issue where `Sink` would sink `readonly` calls past `store release`. For the same reasons described above, this is illegal. Example:
```llvm
%v = call i32 @<!-- -->read_data(ptr @<!-- -->shared_data) ; readonly, argmem only
store atomic i32 1, ptr @<!-- -->done_flag release
br i1 %c, label %use, label %skip ; %v sunk into %use
```
Fortunately, we can adjust our original LICM patch to at least allow the hoisting of readonly calls out of loops, so I'm submitting both changes together (albeit as separate commits) to avoid regressing performance for users relying on that hoist.
---
Patch is 32.94 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/210568.diff
7 Files Affected:
- (modified) llvm/include/llvm/Analysis/MemorySSA.h (+18)
- (modified) llvm/lib/Analysis/AliasAnalysis.cpp (+24)
- (modified) llvm/lib/Analysis/MemorySSA.cpp (+86-4)
- (modified) llvm/lib/Transforms/Scalar/LICM.cpp (+33-3)
- (added) llvm/test/Transforms/LICM/hoist-call-past-release-store.ll (+82)
- (added) llvm/test/Transforms/LICM/hoist-load-past-release-store.ll (+332)
- (added) llvm/test/Transforms/Sink/call-past-release-store.ll (+89)
``````````diff
diff --git a/llvm/include/llvm/Analysis/MemorySSA.h b/llvm/include/llvm/Analysis/MemorySSA.h
index b95a8d0a80e9f..e59f0840a41fe 100644
--- a/llvm/include/llvm/Analysis/MemorySSA.h
+++ b/llvm/include/llvm/Analysis/MemorySSA.h
@@ -1075,6 +1075,22 @@ class MemorySSAWalker {
return getClobberingMemoryAccess(MA, Loc, BAA);
}
+ /// Does the same thing as getClobberingMemoryAccess(MA, BAA), except that
+ /// atomic store defs with release (or weaker) ordering that provably have
+ /// no data effect on the queried access are not treated as clobbers.
+ ///
+ /// Such stores are normally reported as clobbering every escaped location
+ /// (see getSyncEffects) solely because their ordering forbids moving
+ /// program-order-earlier accesses below them. They impose nothing on
+ /// program-order-later reads, so the result of this query is valid ONLY
+ /// for clients that move the queried access to an *earlier* program point
+ /// (hoisting): it must not be used to justify sinking any access below,
+ /// or deleting any access above, one of the skipped stores.
+ ///
+ /// The result is never cached, and cached results are not consulted.
+ virtual MemoryAccess *getClobberingMemoryAccessForHoist(MemoryUseOrDef *,
+ BatchAAResults &) = 0;
+
/// Given a memory access, invalidate anything this walker knows about
/// that access.
/// This API is used by walkers that store information to perform basic cache
@@ -1101,6 +1117,8 @@ class LLVM_ABI DoNothingMemorySSAWalker final : public MemorySSAWalker {
MemoryAccess *getClobberingMemoryAccess(MemoryAccess *,
const MemoryLocation &,
BatchAAResults &) override;
+ MemoryAccess *getClobberingMemoryAccessForHoist(MemoryUseOrDef *,
+ BatchAAResults &) override;
};
/// Iterator base class used to implement const and non-const iterators
diff --git a/llvm/lib/Analysis/AliasAnalysis.cpp b/llvm/lib/Analysis/AliasAnalysis.cpp
index c7984c35e87ae..7837625f9d90a 100644
--- a/llvm/lib/Analysis/AliasAnalysis.cpp
+++ b/llvm/lib/Analysis/AliasAnalysis.cpp
@@ -201,6 +201,21 @@ ModRefInfo AAResults::getModRefInfo(const Instruction *I,
return getModRefInfo(I, Call2, AAQIP);
}
+/// Return true if \p I is an atomic operation whose ordering is stronger
+/// than monotonic, i.e. one that has synchronization effects on memory
+/// beyond its own location.
+static bool hasOrderingStrongerThanMonotonic(const Instruction *I) {
+ if (auto *LI = dyn_cast<LoadInst>(I))
+ return isStrongerThanMonotonic(LI->getOrdering());
+ if (auto *SI = dyn_cast<StoreInst>(I))
+ return isStrongerThanMonotonic(SI->getOrdering());
+ if (auto *RMW = dyn_cast<AtomicRMWInst>(I))
+ return isStrongerThanMonotonic(RMW->getOrdering());
+ if (auto *CX = dyn_cast<AtomicCmpXchgInst>(I))
+ return isStrongerThanMonotonic(CX->getMergedOrdering());
+ return false;
+}
+
ModRefInfo AAResults::getModRefInfo(const Instruction *I, const CallBase *Call2,
AAQueryInfo &AAQI) {
// We may have two calls.
@@ -211,6 +226,15 @@ ModRefInfo AAResults::getModRefInfo(const Instruction *I, const CallBase *Call2,
// If this is a fence, just return ModRef.
if (I->isFenceLike())
return ModRefInfo::ModRef;
+ // An atomic operation stronger than monotonic also synchronizes with other
+ // threads: like the location-based overloads (see getSyncEffects),
+ // conservatively treat it as ordering any memory the call may access, even
+ // memory disjoint from the atomic's own location. Without this, a client
+ // could e.g. sink a read-only call below a release store that publishes
+ // unrelated memory, breaking the publication guarantee.
+ if (hasOrderingStrongerThanMonotonic(I) &&
+ !getMemoryEffects(Call2, AAQI).doesNotAccessMemory())
+ return ModRefInfo::ModRef;
// Otherwise, check if the call modifies or references the
// location this memory access defines. The best we can say
// is that if the call references what this instruction
diff --git a/llvm/lib/Analysis/MemorySSA.cpp b/llvm/lib/Analysis/MemorySSA.cpp
index ebe10b9073a00..f24f68a3c5306 100644
--- a/llvm/lib/Analysis/MemorySSA.cpp
+++ b/llvm/lib/Analysis/MemorySSA.cpp
@@ -273,10 +273,32 @@ static bool areLoadsReorderable(const LoadInst *Use,
template <typename AliasAnalysisType>
static bool
instructionClobbersQuery(const MemoryDef *MD, const MemoryLocation &UseLoc,
- const Instruction *UseInst, AliasAnalysisType &AA) {
+ const Instruction *UseInst, AliasAnalysisType &AA,
+ bool SkipNonAliasingReleaseStores = false) {
Instruction *DefInst = MD->getMemoryInst();
assert(DefInst && "Defining instruction not actually an instruction");
+ // A store with release (or weaker) ordering constrains program-order-earlier
+ // accesses only. It is reported as clobbering escaped locations it cannot
+ // alias (see getSyncEffects) solely so that accesses are not illegally
+ // moved below it or deleted above it. A walk on behalf of a client that
+ // only moves the queried access to an *earlier* program point may opt out
+ // of that ordering component for stores that provably have no data effect
+ // on the queried access; see getClobberingMemoryAccessForHoist.
+ if (SkipNonAliasingReleaseStores) {
+ if (auto *SI = dyn_cast<StoreInst>(DefInst);
+ SI && SI->getOrdering() == AtomicOrdering::Release &&
+ !SI->isVolatile()) {
+ // For a call, the store has no data effect if the call does not access
+ // the store's location; for a load, if the locations cannot alias.
+ if (auto *CB = dyn_cast_or_null<CallBase>(UseInst))
+ return isModOrRefSet(AA.getModRefInfo(CB, MemoryLocation::get(SI)));
+ if (UseLoc.Ptr &&
+ AA.alias(MemoryLocation::get(SI), UseLoc) == AliasResult::NoAlias)
+ return false;
+ }
+ }
+
if (const IntrinsicInst *II = dyn_cast<IntrinsicInst>(DefInst)) {
// These intrinsics will show up as affecting memory, but they are just
// markers, mostly.
@@ -348,6 +370,10 @@ struct UpwardsMemoryQuery {
// The MemoryAccess we actually got called with, used to test local domination
const MemoryAccess *OriginalAccess = nullptr;
bool SkipSelfAccess = false;
+ // Treat release-or-weaker store defs with no data effect on the queried
+ // access as non-clobbering. Only valid for hoisting; results computed with
+ // this set must never be cached. See getClobberingMemoryAccessForHoist.
+ bool SkipNonAliasingReleaseStores = false;
UpwardsMemoryQuery() = default;
@@ -423,7 +449,8 @@ checkClobberSanity(MemoryAccess *Start, MemoryAccess *ClobberAt,
FoundClobber = FoundClobber || MSSA.isLiveOnEntryDef(MD);
if (!FoundClobber) {
BatchAACrossIterationScope _(AA, MAP.MayBeCrossIteration);
- if (instructionClobbersQuery(MD, MAP.Loc, Query.Inst, AA))
+ if (instructionClobbersQuery(MD, MAP.Loc, Query.Inst, AA,
+ Query.SkipNonAliasingReleaseStores))
FoundClobber = true;
}
}
@@ -439,7 +466,8 @@ checkClobberSanity(MemoryAccess *Start, MemoryAccess *ClobberAt,
continue;
BatchAACrossIterationScope _(AA, MAP.MayBeCrossIteration);
- assert(!instructionClobbersQuery(MD, MAP.Loc, Query.Inst, AA) &&
+ assert(!instructionClobbersQuery(MD, MAP.Loc, Query.Inst, AA,
+ Query.SkipNonAliasingReleaseStores) &&
"Found clobber before reaching ClobberAt!");
continue;
}
@@ -575,7 +603,8 @@ class ClobberWalker {
return {Current, true};
BatchAACrossIterationScope _(*AA, Desc.MayBeCrossIteration);
- if (instructionClobbersQuery(MD, Desc.Loc, Query->Inst, *AA))
+ if (instructionClobbersQuery(MD, Desc.Loc, Query->Inst, *AA,
+ Query->SkipNonAliasingReleaseStores))
return {MD, true};
}
}
@@ -1007,6 +1036,12 @@ class MemorySSA::ClobberWalkerBase {
MemoryAccess *getClobberingMemoryAccessBase(MemoryAccess *, BatchAAResults &,
unsigned &, bool,
bool UseInvariantGroup = true);
+ // Uncached walk that skips non-aliasing release-or-weaker store defs; only
+ // valid for hoisting the queried access. See
+ // MemorySSAWalker::getClobberingMemoryAccessForHoist.
+ MemoryAccess *getClobberingMemoryAccessForHoistBase(MemoryUseOrDef *,
+ BatchAAResults &,
+ unsigned &);
};
/// A MemorySSAWalker that does AA walks to disambiguate accesses. It no
@@ -1049,6 +1084,13 @@ class MemorySSA::CachingWalker final : public MemorySSAWalker {
return getClobberingMemoryAccess(MA, Loc, BAA, UpwardWalkLimit);
}
+ MemoryAccess *getClobberingMemoryAccessForHoist(MemoryUseOrDef *MA,
+ BatchAAResults &BAA) override {
+ unsigned UpwardWalkLimit = MaxCheckLimit;
+ return Walker->getClobberingMemoryAccessForHoistBase(MA, BAA,
+ UpwardWalkLimit);
+ }
+
void invalidateInfo(MemoryAccess *MA) override {
if (auto *MUD = dyn_cast<MemoryUseOrDef>(MA))
MUD->resetOptimized();
@@ -1087,6 +1129,13 @@ class MemorySSA::SkipSelfWalker final : public MemorySSAWalker {
return getClobberingMemoryAccess(MA, Loc, BAA, UpwardWalkLimit);
}
+ MemoryAccess *getClobberingMemoryAccessForHoist(MemoryUseOrDef *MA,
+ BatchAAResults &BAA) override {
+ unsigned UpwardWalkLimit = MaxCheckLimit;
+ return Walker->getClobberingMemoryAccessForHoistBase(MA, BAA,
+ UpwardWalkLimit);
+ }
+
void invalidateInfo(MemoryAccess *MA) override {
if (auto *MUD = dyn_cast<MemoryUseOrDef>(MA))
MUD->resetOptimized();
@@ -2612,6 +2661,34 @@ MemoryAccess *MemorySSA::ClobberWalkerBase::getClobberingMemoryAccessBase(
return Result;
}
+MemoryAccess *
+MemorySSA::ClobberWalkerBase::getClobberingMemoryAccessForHoistBase(
+ MemoryUseOrDef *StartingAccess, BatchAAResults &BAA,
+ unsigned &UpwardWalkLimit) {
+ const Instruction *I = StartingAccess->getMemoryInst();
+
+ // We can't sanely do anything with a fence; conservatively return the
+ // access itself, like the location-based walk.
+ if (!isa<CallBase>(I) && I->isFenceLike())
+ return StartingAccess;
+
+ if (isUseTriviallyOptimizableToLiveOnEntry(BAA, I))
+ return MSSA->getLiveOnEntryDef();
+
+ MemoryAccess *DefiningAccess = StartingAccess->getDefiningAccess();
+ if (MSSA->isLiveOnEntryDef(DefiningAccess))
+ return DefiningAccess;
+
+ UpwardsMemoryQuery Q(I, StartingAccess);
+ Q.SkipNonAliasingReleaseStores = true;
+
+ // Deliberately bypass the isOptimized()/setOptimized() cache in both
+ // directions: results computed while skipping release stores are only
+ // valid for hoisting the queried access and must never be visible to any
+ // other client.
+ return Walker.findClobber(BAA, DefiningAccess, Q, UpwardWalkLimit);
+}
+
MemoryAccess *
DoNothingMemorySSAWalker::getClobberingMemoryAccess(MemoryAccess *MA,
BatchAAResults &) {
@@ -2620,6 +2697,11 @@ DoNothingMemorySSAWalker::getClobberingMemoryAccess(MemoryAccess *MA,
return MA;
}
+MemoryAccess *DoNothingMemorySSAWalker::getClobberingMemoryAccessForHoist(
+ MemoryUseOrDef *MA, BatchAAResults &) {
+ return MA->getDefiningAccess();
+}
+
MemoryAccess *DoNothingMemorySSAWalker::getClobberingMemoryAccess(
MemoryAccess *StartingAccess, const MemoryLocation &, BatchAAResults &) {
if (auto *Use = dyn_cast<MemoryUseOrDef>(StartingAccess))
diff --git a/llvm/lib/Transforms/Scalar/LICM.cpp b/llvm/lib/Transforms/Scalar/LICM.cpp
index c7e29b70662d8..7501b28363af9 100644
--- a/llvm/lib/Transforms/Scalar/LICM.cpp
+++ b/llvm/lib/Transforms/Scalar/LICM.cpp
@@ -2373,10 +2373,40 @@ static bool pointerInvalidatedByLoop(MemorySSA *MSSA, MemoryUse *MU,
// if the memory loaded is the phi node
BatchAAResults BAA(MSSA->getAA());
+ // Fast path: this query is answered from MemorySSA's cache (uses are
+ // optimized at construction time), unlike the uncached
+ // getClobberingMemoryAccessForHoist walk below, which we therefore only
+ // run if this conservative verdict blocks hoisting.
MemoryAccess *Source = getClobberingMemoryAccess(*MSSA, BAA, Flags, MU);
- return !MSSA->isLiveOnEntryDef(Source) &&
- CurLoop->contains(Source->getBlock()) &&
- !(InvariantGroup && Source->getBlock() == CurLoop->getHeader() && isa<MemoryPhi>(Source));
+ bool Invalidated =
+ !MSSA->isLiveOnEntryDef(Source) &&
+ CurLoop->contains(Source->getBlock()) &&
+ !(InvariantGroup && Source->getBlock() == CurLoop->getHeader() &&
+ isa<MemoryPhi>(Source));
+
+ // The walker treats release stores as clobbers of any escaped location
+ // (see getSyncEffects) because their ordering forbids moving
+ // program-order-earlier accesses below them; that blocks hoisting loads
+ // and read-only calls out of loops that publish unrelated data. Hoisting
+ // only moves the access to an earlier program point, which a
+ // release-or-weaker store does not constrain, so retry the walk with
+ // such stores skipped when they provably have no data effect on the
+ // access. This must only be done when hoisting: the sink path below
+ // stays conservative, as sinking below a release store is illegal.
+ if (Invalidated && !Flags.tooManyClobberingCalls()) {
+ auto *LI = dyn_cast<LoadInst>(&I);
+ if ((LI && LI->isUnordered()) || (!LI && isa<CallBase>(&I))) {
+ Source = MSSA->getWalker()->getClobberingMemoryAccessForHoist(MU, BAA);
+ Flags.incrementClobberingCalls();
+ Invalidated =
+ !MSSA->isLiveOnEntryDef(Source) &&
+ CurLoop->contains(Source->getBlock()) &&
+ !(InvariantGroup && Source->getBlock() == CurLoop->getHeader() &&
+ isa<MemoryPhi>(Source));
+ }
+ }
+
+ return Invalidated;
}
// For sinking, we'd need to check all Defs below this use. The getClobbering
diff --git a/llvm/test/Transforms/LICM/hoist-call-past-release-store.ll b/llvm/test/Transforms/LICM/hoist-call-past-release-store.ll
new file mode 100644
index 0000000000000..2a296ad70c80e
--- /dev/null
+++ b/llvm/test/Transforms/LICM/hoist-call-past-release-store.ll
@@ -0,0 +1,82 @@
+; RUN: opt -passes='loop-mssa(licm)' -S < %s | FileCheck %s
+;
+; A read-only call, like a load, may be hoisted above a release store to
+; memory it does not access: the release ordering constrains only
+; program-order-earlier accesses. Keep seq_cst stores and release RMWs as
+; conservative barriers.
+
+ at flag = global i32 0
+ at data = global i32 0
+
+declare i32 @read_data(ptr) nounwind willreturn memory(argmem: read)
+
+define i32 @hoist_call_past_release_store(i32 %n) {
+; CHECK-LABEL: @hoist_call_past_release_store(
+; CHECK: entry:
+; CHECK-NEXT: [[V:%.*]] = call i32 @read_data(ptr @data)
+; CHECK-NEXT: br label %loop
+; CHECK: loop:
+; CHECK-NOT: call i32 @read_data
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+ %sum = phi i32 [ 0, %entry ], [ %sum.next, %loop ]
+ %v = call i32 @read_data(ptr @data)
+ %sum.next = add i32 %sum, %v
+ store atomic i32 1, ptr @flag release, align 4
+ %i.next = add nuw nsw i32 %i, 1
+ %cmp = icmp slt i32 %i.next, %n
+ br i1 %cmp, label %loop, label %exit
+
+exit:
+ ret i32 %sum.next
+}
+
+; Negative: seq_cst stores stay conservative.
+define i32 @no_hoist_call_past_seq_cst_store(i32 %n) {
+; CHECK-LABEL: @no_hoist_call_past_seq_cst_store(
+; CHECK: loop:
+; CHECK: call i32 @read_data(ptr @data)
+; CHECK-NEXT: {{%.*}} = add
+; CHECK-NEXT: store atomic i32 1, ptr @flag seq_cst
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+ %sum = phi i32 [ 0, %entry ], [ %sum.next, %loop ]
+ %v = call i32 @read_data(ptr @data)
+ %sum.next = add i32 %sum, %v
+ store atomic i32 1, ptr @flag seq_cst, align 4
+ %i.next = add nuw nsw i32 %i, 1
+ %cmp = icmp slt i32 %i.next, %n
+ br i1 %cmp, label %loop, label %exit
+
+exit:
+ ret i32 %sum.next
+}
+
+; Negative: the call may read the location the release store writes; that is
+; a real data dependence, not just ordering.
+define i32 @no_hoist_call_reads_flag(i32 %n) {
+; CHECK-LABEL: @no_hoist_call_reads_flag(
+; CHECK: loop:
+; CHECK: call i32 @read_data(ptr @flag)
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+ %sum = phi i32 [ 0, %entry ], [ %sum.next, %loop ]
+ %v = call i32 @read_data(ptr @flag)
+ %sum.next = add i32 %sum, %v
+ store atomic i32 1, ptr @flag release, align 4
+ %i.next = add nuw nsw i32 %i, 1
+ %cmp = icmp slt i32 %i.next, %n
+ br i1 %cmp, label %loop, label %exit
+
+exit:
+ ret i32 %sum.next
+}
diff --git a/llvm/test/Transforms/LICM/hoist-load-past-release-store.ll b/llvm/test/Transforms/LICM/hoist-load-past-release-store.ll
new file mode 100644
index 0000000000000..4bd6f67fc4844
--- /dev/null
+++ b/llvm/test/Transforms/LICM/hoist-load-past-release-store.ll
@@ -0,0 +1,332 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -passes='loop-mssa(licm)' -S < %s | FileCheck %s
+
+; A store with release (or weaker) ordering imposes no constraint on
+; program-order-later reads of unrelated memory: moving a load from after a
+; release store to before it is explicitly permitted by the memory model.
+; Check that LICM hoists loop-invariant loads out of loops whose body contains
+; a release store to provably non-aliasing memory.
+
+ at flag = global i32 0
+ at data = global i32 0
+
+; The load of @data is loop invariant; the release store to @flag must not
+; block hoisting it into the preheader.
+define i32 @hoist_past_release_store(i32 %n) {
+; CHECK-LABEL: @hoist_past_release_store(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: [[VAL:%.*]] = load i32, ptr @data, align 4
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[I_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[SUM:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[SUM_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[SUM_NEXT]] = add i32 [[SUM]], [[VAL]]
+; CHECK-NEXT: store atomic i32 1, ptr @flag release, align 4
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i32 [[I]], 1
+; CHECK-NEXT: [[CMP:%.*]] = icmp slt i32 [[I_NEXT]], [[N:%.*]]
+; CHECK-NEXT: br i1 [[CMP]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK: exit:
+; CHECK-NEXT: [[SUM_NEXT_LCSSA:%.*]] = phi i32 [ [[SUM_NEXT]], [[LOOP]] ]
+; CHECK-NEXT: ret i32 [[SUM_NEXT_LCSSA]]
+;
+entry:
+ br label %loop
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+ %sum = phi i32 [ 0, %entry ], [ %sum.next, %loop ]
+ %val = load i32, ptr @data, align 4
+ %sum.next = add i32 %sum, %val
+ store atomic i32 1, ptr @flag release, align 4
+ %i.next = add nuw nsw i32 %i, 1
+ %cmp = icmp slt i32 %i.next, %n
+ br i1 %cmp, label %loop, label %exit
+
+exit:
+ ret i32 %sum.next
+}
+
+; Same for a monotonic store.
+define i32 @hoist_past_monotonic_store(i32 %n) {
+; CHECK-LABEL: @hoist_past_monotonic_store(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: [[VAL:%.*]] = load i32, ptr @data, align 4
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[I_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[SUM:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[SUM_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[SUM_NEXT]] = add i32 [[SUM]], [[VAL]]
+; CHECK-NEXT: store atomic i32 1, ptr @flag monotonic, align 4
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i32 [[I]], 1
+; CHECK-NEXT: [[CMP:%.*]] = icmp slt i32 [[I_NEXT]], [[N:%.*]]
+; CHECK-NEXT: br i1 [[CMP]], label [[LOOP]], label [[EXIT:%.*]]
+; CHECK: exit:
+; CHECK-NEXT: ...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/210568
More information about the llvm-commits
mailing list