[flang-commits] [flang] [llvm] [AtomicExpand] Let targets keep the release fence out of the reservation (PR #214867)
Josef Schlehofer via flang-commits
flang-commits at lists.llvm.org
Wed Sep 23 05:55:45 PDT 2026
https://github.com/BKPepe updated https://github.com/llvm/llvm-project/pull/214867
>From ad9ca22b2a5bc03eefa68b032071a28128806122 Mon Sep 17 00:00:00 2001
From: Josef Schlehofer <pepe at bloodkings.eu>
Date: Sat, 8 Aug 2026 00:31:56 +0200
Subject: [PATCH 1/3] [AtomicExpand] Let targets keep the release fence out of
the reservation
expandAtomicCmpXchg sinks the leading fence into the conditional
cmpxchg.fencedstore block, which places it between the load-linked and the
store-conditional. For the strong form that is harmless: if the
store-conditional fails, the retry re-reserves after the fence and the loop
still converges. The weak form has no retry, so where a fence can clear the
reservation it can never succeed.
Add fenceClearsLoadLinkedReservation(), defaulting to false, and hoist the
fence for a weak cmpxchg only on targets that say yes. PowerPC does: the
Power ISA lets an implementation drop a reservation for reasons of its own,
and e500v2 does so for a sync between the lwarx and the stwcx., which is
where this was found. Every compare_exchange_weak with a release ordering
spins there forever, and since fetch_update in Rust's core is a weak retry
loop, async runtimes hang on the platform.
Hoisting moves the fence ahead of the comparison, so it also executes on the
path where the comparison fails and no store follows. The fence is only
needed to provide the release ordering of a successful store; executing it
on an attempt that fails before the store is permitted and has no additional
ordering effect. It adds the cost of the fence on that path and nothing
else -- no extra synchronisation, store or loop.
For targets where the hook stays false the UseUnconditionalReleaseBarrier
condition is unchanged from the previous implementation, so their fence
placement and codegen are preserved. ARM is one: it emits a DMB in the sunk
position and its monitor survives one, so it has no reason to opt in.
Co-Authored-By: Claude Opus 5 <noreply at anthropic.com>
---
llvm/include/llvm/CodeGen/TargetLowering.h | 7 +
llvm/lib/CodeGen/AtomicExpandPass.cpp | 12 +-
llvm/lib/Target/PowerPC/PPCISelLowering.h | 7 +
llvm/test/CodeGen/PowerPC/atomics.ll | 2 +-
.../PowerPC/weak-cmpxchg-fence-placement.ll | 166 ++++++++++++++++++
5 files changed, 192 insertions(+), 2 deletions(-)
create mode 100644 llvm/test/Transforms/AtomicExpand/PowerPC/weak-cmpxchg-fence-placement.ll
diff --git a/llvm/include/llvm/CodeGen/TargetLowering.h b/llvm/include/llvm/CodeGen/TargetLowering.h
index 69f5e0e4e30111..ff4797ce768308 100644
--- a/llvm/include/llvm/CodeGen/TargetLowering.h
+++ b/llvm/include/llvm/CodeGen/TargetLowering.h
@@ -2313,6 +2313,13 @@ class LLVM_ABI TargetLoweringBase {
return false;
}
+ /// Whether a fence placed between the load-linked and the store-conditional
+ /// can clear the reservation on this target. When it can, AtomicExpandPass
+ /// must not sink the leading fence of a weak cmpxchg into the reservation
+ /// window: the store-conditional would fail, and a weak cmpxchg has no retry
+ /// to re-reserve and recover with. Defaults to false.
+ virtual bool fenceClearsLoadLinkedReservation() const { return false; }
+
/// Whether AtomicExpandPass should automatically insert a seq_cst trailing
/// fence without reducing the ordering for this atomic store. Defaults to
/// false.
diff --git a/llvm/lib/CodeGen/AtomicExpandPass.cpp b/llvm/lib/CodeGen/AtomicExpandPass.cpp
index f2ffa40030cc0c..707800e96e1f20 100644
--- a/llvm/lib/CodeGen/AtomicExpandPass.cpp
+++ b/llvm/lib/CodeGen/AtomicExpandPass.cpp
@@ -1498,7 +1498,17 @@ bool AtomicExpandImpl::expandAtomicCmpXchg(AtomicCmpXchgInst *CI) {
// There's no overhead for sinking the release barrier in a weak cmpxchg, so
// do it even on minsize.
- bool UseUnconditionalReleaseBarrier = F->hasMinSize() && !CI->isWeak();
+ //
+ // Except where a fence can clear the reservation. Sinking puts the fence
+ // between the load-linked and the store-conditional, and there the
+ // store-conditional fails; the strong form recovers because its retry
+ // re-reserves afterwards, but a weak cmpxchg has no retry and so could never
+ // succeed. Hoisting is valid because the fence is only needed to provide the
+ // release ordering of a successful store, and executing it on an attempt
+ // that fails before the store has no additional ordering effect.
+ bool UseUnconditionalReleaseBarrier =
+ (F->hasMinSize() && !CI->isWeak()) ||
+ (CI->isWeak() && TLI->fenceClearsLoadLinkedReservation());
// Given: cmpxchg some_op iN* %addr, iN %desired, iN %new success_ord fail_ord
//
diff --git a/llvm/lib/Target/PowerPC/PPCISelLowering.h b/llvm/lib/Target/PowerPC/PPCISelLowering.h
index fc0e776521e084..8086e56a796f79 100644
--- a/llvm/lib/Target/PowerPC/PPCISelLowering.h
+++ b/llvm/lib/Target/PowerPC/PPCISelLowering.h
@@ -338,6 +338,13 @@ namespace llvm {
return true;
}
+ /// The Power ISA lets an implementation clear a reservation for reasons of
+ /// its own, and e500v2 does so for a sync between the lwarx and the
+ /// stwcx., which leaves a weak cmpxchg unable to ever succeed. Gating on
+ /// isE500() would miss generic powerpc builds, where nothing enables it;
+ /// the cost elsewhere is one fence on the comparison-failed path.
+ bool fenceClearsLoadLinkedReservation() const override { return true; }
+
Value *emitLoadLinked(IRBuilderBase &Builder, Type *ValueTy, Value *Addr,
AtomicOrdering Ord) const override;
diff --git a/llvm/test/CodeGen/PowerPC/atomics.ll b/llvm/test/CodeGen/PowerPC/atomics.ll
index af2b8646d18ae0..30897e06cbea1c 100644
--- a/llvm/test/CodeGen/PowerPC/atomics.ll
+++ b/llvm/test/CodeGen/PowerPC/atomics.ll
@@ -310,12 +310,12 @@ define i64 @cas_weak_i64_release_monotonic(ptr %mem) {
;
; PPC64-LABEL: cas_weak_i64_release_monotonic:
; PPC64: # %bb.0: # %cmpxchg.start
+; PPC64-NEXT: lwsync
; PPC64-NEXT: mr r4, r3
; PPC64-NEXT: ldarx r3, 0, r3
; PPC64-NEXT: cmpldi r3, 0
; PPC64-NEXT: bnelr- cr0
; PPC64-NEXT: # %bb.1: # %cmpxchg.fencedstore
-; PPC64-NEXT: lwsync
; PPC64-NEXT: li r5, 1
; PPC64-NEXT: stdcx. r5, 0, r4
; PPC64-NEXT: blr
diff --git a/llvm/test/Transforms/AtomicExpand/PowerPC/weak-cmpxchg-fence-placement.ll b/llvm/test/Transforms/AtomicExpand/PowerPC/weak-cmpxchg-fence-placement.ll
new file mode 100644
index 00000000000000..1a0fd06eb03a36
--- /dev/null
+++ b/llvm/test/Transforms/AtomicExpand/PowerPC/weak-cmpxchg-fence-placement.ll
@@ -0,0 +1,166 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 5
+; RUN: opt -S -mtriple=powerpc-unknown-linux-musl \
+; RUN: -passes='require<libcall-lowering-info>,atomic-expand' %s | FileCheck %s
+
+; The release barrier for a weak cmpxchg belongs before the load-linked, not
+; between it and the store-conditional. A barrier inside the reservation window
+; can clear the reservation, and a weak cmpxchg has no retry to recover with, so
+; sinking it there leaves an operation that can never succeed. The strong form
+; keeps the sunk barrier: its retry re-reserves afterwards, so it converges
+; either way.
+
+define i1 @weak_acq_rel(ptr %addr) {
+; CHECK-LABEL: define i1 @weak_acq_rel(
+; CHECK-SAME: ptr [[ADDR:%.*]]) {
+; CHECK-NEXT: call void @llvm.ppc.lwsync()
+; CHECK-NEXT: br label %[[CMPXCHG_START:.*]]
+; CHECK: [[CMPXCHG_START]]:
+; CHECK-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
+; CHECK-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
+; CHECK-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0:![0-9]+]]
+; CHECK: [[CMPXCHG_FENCEDSTORE]]:
+; CHECK-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
+; CHECK: [[CMPXCHG_TRYSTORE]]:
+; CHECK-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ]
+; CHECK-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
+; CHECK-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
+; CHECK-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
+; CHECK-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_FAILURE:.*]], !prof [[PROF0]]
+; CHECK: [[CMPXCHG_RELEASEDLOAD:.*:]]
+; CHECK-NEXT: unreachable
+; CHECK: [[CMPXCHG_SUCCESS]]:
+; CHECK-NEXT: call void @llvm.ppc.lwsync()
+; CHECK-NEXT: br label %[[CMPXCHG_END:.*]]
+; CHECK: [[CMPXCHG_NOSTORE]]:
+; CHECK-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ]
+; CHECK-NEXT: br label %[[CMPXCHG_FAILURE]]
+; CHECK: [[CMPXCHG_FAILURE]]:
+; CHECK-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ], [ [[LOADED_TRYSTORE]], %[[CMPXCHG_TRYSTORE]] ]
+; CHECK-NEXT: call void @llvm.ppc.lwsync()
+; CHECK-NEXT: br label %[[CMPXCHG_END]]
+; CHECK: [[CMPXCHG_END]]:
+; CHECK-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
+; CHECK-NEXT: [[SUCCESS1:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
+; CHECK-NEXT: ret i1 [[SUCCESS1]]
+;
+ %pair = cmpxchg weak ptr %addr, i32 0, i32 1 acq_rel acquire
+ %ok = extractvalue { i32, i1 } %pair, 1
+ ret i1 %ok
+}
+
+define i1 @weak_release(ptr %addr) {
+; CHECK-LABEL: define i1 @weak_release(
+; CHECK-SAME: ptr [[ADDR:%.*]]) {
+; CHECK-NEXT: call void @llvm.ppc.lwsync()
+; CHECK-NEXT: br label %[[CMPXCHG_START:.*]]
+; CHECK: [[CMPXCHG_START]]:
+; CHECK-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
+; CHECK-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
+; CHECK-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0]]
+; CHECK: [[CMPXCHG_FENCEDSTORE]]:
+; CHECK-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
+; CHECK: [[CMPXCHG_TRYSTORE]]:
+; CHECK-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ]
+; CHECK-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
+; CHECK-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
+; CHECK-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
+; CHECK-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_FAILURE:.*]], !prof [[PROF0]]
+; CHECK: [[CMPXCHG_RELEASEDLOAD:.*:]]
+; CHECK-NEXT: unreachable
+; CHECK: [[CMPXCHG_SUCCESS]]:
+; CHECK-NEXT: br label %[[CMPXCHG_END:.*]]
+; CHECK: [[CMPXCHG_NOSTORE]]:
+; CHECK-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ]
+; CHECK-NEXT: br label %[[CMPXCHG_FAILURE]]
+; CHECK: [[CMPXCHG_FAILURE]]:
+; CHECK-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ], [ [[LOADED_TRYSTORE]], %[[CMPXCHG_TRYSTORE]] ]
+; CHECK-NEXT: br label %[[CMPXCHG_END]]
+; CHECK: [[CMPXCHG_END]]:
+; CHECK-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
+; CHECK-NEXT: [[SUCCESS1:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
+; CHECK-NEXT: ret i1 [[SUCCESS1]]
+;
+ %pair = cmpxchg weak ptr %addr, i32 0, i32 1 release monotonic
+ %ok = extractvalue { i32, i1 } %pair, 1
+ ret i1 %ok
+}
+
+define i1 @strong_acq_rel(ptr %addr) {
+; CHECK-LABEL: define i1 @strong_acq_rel(
+; CHECK-SAME: ptr [[ADDR:%.*]]) {
+; CHECK-NEXT: br label %[[CMPXCHG_START:.*]]
+; CHECK: [[CMPXCHG_START]]:
+; CHECK-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
+; CHECK-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
+; CHECK-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0]]
+; CHECK: [[CMPXCHG_FENCEDSTORE]]:
+; CHECK-NEXT: call void @llvm.ppc.lwsync()
+; CHECK-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
+; CHECK: [[CMPXCHG_TRYSTORE]]:
+; CHECK-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ], [ [[LARX1:%.*]], %[[CMPXCHG_RELEASEDLOAD:.*]] ]
+; CHECK-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
+; CHECK-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
+; CHECK-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
+; CHECK-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_RELEASEDLOAD]], !prof [[PROF0]]
+; CHECK: [[CMPXCHG_RELEASEDLOAD]]:
+; CHECK-NEXT: [[LARX1]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
+; CHECK-NEXT: [[SHOULD_STORE2:%.*]] = icmp eq i32 [[LARX1]], 0
+; CHECK-NEXT: br i1 [[SHOULD_STORE2]], label %[[CMPXCHG_TRYSTORE]], label %[[CMPXCHG_NOSTORE]], !prof [[PROF0]]
+; CHECK: [[CMPXCHG_SUCCESS]]:
+; CHECK-NEXT: call void @llvm.ppc.lwsync()
+; CHECK-NEXT: br label %[[CMPXCHG_END:.*]]
+; CHECK: [[CMPXCHG_NOSTORE]]:
+; CHECK-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ], [ [[LARX1]], %[[CMPXCHG_RELEASEDLOAD]] ]
+; CHECK-NEXT: br label %[[CMPXCHG_FAILURE:.*]]
+; CHECK: [[CMPXCHG_FAILURE]]:
+; CHECK-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ]
+; CHECK-NEXT: call void @llvm.ppc.lwsync()
+; CHECK-NEXT: br label %[[CMPXCHG_END]]
+; CHECK: [[CMPXCHG_END]]:
+; CHECK-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
+; CHECK-NEXT: [[SUCCESS3:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
+; CHECK-NEXT: ret i1 [[SUCCESS3]]
+;
+ %pair = cmpxchg ptr %addr, i32 0, i32 1 acq_rel acquire
+ %ok = extractvalue { i32, i1 } %pair, 1
+ ret i1 %ok
+}
+
+define i1 @weak_monotonic(ptr %addr) {
+; CHECK-LABEL: define i1 @weak_monotonic(
+; CHECK-SAME: ptr [[ADDR:%.*]]) {
+; CHECK-NEXT: br label %[[CMPXCHG_START:.*]]
+; CHECK: [[CMPXCHG_START]]:
+; CHECK-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
+; CHECK-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
+; CHECK-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0]]
+; CHECK: [[CMPXCHG_FENCEDSTORE]]:
+; CHECK-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
+; CHECK: [[CMPXCHG_TRYSTORE]]:
+; CHECK-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ]
+; CHECK-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
+; CHECK-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
+; CHECK-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
+; CHECK-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_FAILURE:.*]], !prof [[PROF0]]
+; CHECK: [[CMPXCHG_RELEASEDLOAD:.*:]]
+; CHECK-NEXT: unreachable
+; CHECK: [[CMPXCHG_SUCCESS]]:
+; CHECK-NEXT: br label %[[CMPXCHG_END:.*]]
+; CHECK: [[CMPXCHG_NOSTORE]]:
+; CHECK-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ]
+; CHECK-NEXT: br label %[[CMPXCHG_FAILURE]]
+; CHECK: [[CMPXCHG_FAILURE]]:
+; CHECK-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ], [ [[LOADED_TRYSTORE]], %[[CMPXCHG_TRYSTORE]] ]
+; CHECK-NEXT: br label %[[CMPXCHG_END]]
+; CHECK: [[CMPXCHG_END]]:
+; CHECK-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
+; CHECK-NEXT: [[SUCCESS1:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
+; CHECK-NEXT: ret i1 [[SUCCESS1]]
+;
+ %pair = cmpxchg weak ptr %addr, i32 0, i32 1 monotonic monotonic
+ %ok = extractvalue { i32, i1 } %pair, 1
+ ret i1 %ok
+}
+;.
+; CHECK: [[PROF0]] = !{!"branch_weights", i32 1048575, i32 1}
+;.
>From a63943f870bf7d6a27651068fe445787a3fed423 Mon Sep 17 00:00:00 2001
From: Josef Schlehofer <pepe.schlehofer at gmail.com>
Date: Tue, 22 Sep 2026 21:40:21 +0200
Subject: [PATCH 2/3] fixup! [AtomicExpand] Let targets keep the release fence
out of the reservation
Make the reservation-clearing fence a per-CPU property, as requested in
review. Add the PowerPC subtarget feature fence-keeps-reservation and set
it in P8AdditionalFeatures, so pwr8 and later and ppc64le, the
powerpc64le default, keep the fence in the reservation window. CPUs
without it, such as ppc, ppc64 and e500, get the fence ahead of the
larx. The tests cover both placements.
---
flang/test/Lower/target-features-ppc.f90 | 4 +-
llvm/lib/Target/PowerPC/PPC.td | 13 +-
llvm/lib/Target/PowerPC/PPCISelLowering.cpp | 4 +
llvm/lib/Target/PowerPC/PPCISelLowering.h | 10 +-
llvm/test/CodeGen/PowerPC/atomics.ll | 37 ++-
.../PowerPC/weak-cmpxchg-fence-placement.ll | 293 +++++++++++++-----
6 files changed, 270 insertions(+), 91 deletions(-)
diff --git a/flang/test/Lower/target-features-ppc.f90 b/flang/test/Lower/target-features-ppc.f90
index 1c3429e6936a02..9826c96df4cf81 100644
--- a/flang/test/Lower/target-features-ppc.f90
+++ b/flang/test/Lower/target-features-ppc.f90
@@ -7,9 +7,9 @@
! ALL: fir.target_cpu = "pwr10"
! FEATURE: fir.target_features = #llvm.target_features<[
-! FEATURE: "+64bit-support", "+allow-unaligned-fp-access", "+altivec", "+bpermd", "+cmpb", "+crbits", "+crypto", "+direct-move", "+extdiv", "+fast-MFLR", "+fcpsgn", "+fpcvt", "+fprnd", "+fpu", "+fre", "+fres", "+frsqrte", "+frsqrtes", "+fsqrt", "+fuse-add-logical", "+fuse-arith-add", "+fuse-logical", "+fuse-logical-add", "+fuse-sha3", "+fuse-store", "+fusion", "+hard-float", "+icbt", "+isa-v206-instructions", "+isa-v207-instructions", "+isa-v30-instructions", "+isa-v31-instructions", "+isel", "+ldbrx", "+lfiwax", "+mfocrf", "+mma", "+paired-vector-memops", "+partword-atomics", "+pcrelative-memops", "+popcntd", "+power10-vector", "+power8-altivec", "+power8-vector", "+power9-altivec", "+power9-vector", "+ppc-postra-sched", "+ppc-prera-sched", "+predictable-select-expensive", "+prefix-instrs", "+quadword-atomics", "+recipprec", "+stfiwx", "+two-const-nr", "+vsx"
+! FEATURE: "+64bit-support", "+allow-unaligned-fp-access", "+altivec", "+bpermd", "+cmpb", "+crbits", "+crypto", "+direct-move", "+extdiv", "+fast-MFLR", "+fcpsgn", "+fence-keeps-reservation", "+fpcvt", "+fprnd", "+fpu", "+fre", "+fres", "+frsqrte", "+frsqrtes", "+fsqrt", "+fuse-add-logical", "+fuse-arith-add", "+fuse-logical", "+fuse-logical-add", "+fuse-sha3", "+fuse-store", "+fusion", "+hard-float", "+icbt", "+isa-v206-instructions", "+isa-v207-instructions", "+isa-v30-instructions", "+isa-v31-instructions", "+isel", "+ldbrx", "+lfiwax", "+mfocrf", "+mma", "+paired-vector-memops", "+partword-atomics", "+pcrelative-memops", "+popcntd", "+power10-vector", "+power8-altivec", "+power8-vector", "+power9-altivec", "+power9-vector", "+ppc-postra-sched", "+ppc-prera-sched", "+predictable-select-expensive", "+prefix-instrs", "+quadword-atomics", "+recipprec", "+stfiwx", "+two-const-nr", "+vsx"
! FEATURE: ]>
! BOTH: fir.target_features = #llvm.target_features<[
-! BOTH: "+64bit-support", "+allow-unaligned-fp-access", "+altivec", "+bpermd", "+cmpb", "+crbits", "+crypto", "+direct-move", "+extdiv", "+fast-MFLR", "+fcpsgn", "+fpcvt", "+fprnd", "+fpu", "+fre", "+fres", "+frsqrte", "+frsqrtes", "+fsqrt", "+fuse-add-logical", "+fuse-arith-add", "+fuse-logical", "+fuse-logical-add", "+fuse-sha3", "+fuse-store", "+fusion", "+hard-float", "+icbt", "+isa-v206-instructions", "+isa-v207-instructions", "+isa-v30-instructions", "+isa-v31-instructions", "+isel", "+ldbrx", "+lfiwax", "+mfocrf", "+mma", "+paired-vector-memops", "+partword-atomics", "+pcrelative-memops", "+popcntd", "+power10-vector", "+power8-altivec", "+power8-vector", "+power9-altivec", "+power9-vector", "+ppc-postra-sched", "+ppc-prera-sched", "+predictable-select-expensive", "+prefix-instrs", "+privileged", "+quadword-atomics", "+recipprec", "+stfiwx", "+two-const-nr", "+vsx"
+! BOTH: "+64bit-support", "+allow-unaligned-fp-access", "+altivec", "+bpermd", "+cmpb", "+crbits", "+crypto", "+direct-move", "+extdiv", "+fast-MFLR", "+fcpsgn", "+fence-keeps-reservation", "+fpcvt", "+fprnd", "+fpu", "+fre", "+fres", "+frsqrte", "+frsqrtes", "+fsqrt", "+fuse-add-logical", "+fuse-arith-add", "+fuse-logical", "+fuse-logical-add", "+fuse-sha3", "+fuse-store", "+fusion", "+hard-float", "+icbt", "+isa-v206-instructions", "+isa-v207-instructions", "+isa-v30-instructions", "+isa-v31-instructions", "+isel", "+ldbrx", "+lfiwax", "+mfocrf", "+mma", "+paired-vector-memops", "+partword-atomics", "+pcrelative-memops", "+popcntd", "+power10-vector", "+power8-altivec", "+power8-vector", "+power9-altivec", "+power9-vector", "+ppc-postra-sched", "+ppc-prera-sched", "+predictable-select-expensive", "+prefix-instrs", "+privileged", "+quadword-atomics", "+recipprec", "+stfiwx", "+two-const-nr", "+vsx"
! BOTH: ]>
diff --git a/llvm/lib/Target/PowerPC/PPC.td b/llvm/lib/Target/PowerPC/PPC.td
index ba2a4e6a9695ce..312380297d80c8 100644
--- a/llvm/lib/Target/PowerPC/PPC.td
+++ b/llvm/lib/Target/PowerPC/PPC.td
@@ -177,6 +177,16 @@ def FeaturePartwordAtomic : SubtargetFeature<"partword-atomics",
def FeatureQuadwordAtomic : SubtargetFeature<"quadword-atomics",
"HasQuadwordAtomics", "true",
"Enable lqarx and stqcx.">;
+// The Power ISA lets an implementation clear a reservation for reasons of its
+// own, and some clear it when a sync or lwsync is executed between the larx
+// and the stcx. That leaves a weak cmpxchg whose release fence is placed there
+// unable to succeed, so AtomicExpand only places it there on processors with
+// this feature. All others, including ppc, ppc32 and ppc64, get the fence
+// ahead of the larx.
+def FeatureFenceKeepsReservation :
+ SubtargetFeature<"fence-keeps-reservation", "FenceKeepsReservation", "true",
+ "A sync or lwsync between larx and stcx. does not clear "
+ "the reservation", [], InlineIgnore>;
def FeatureInvariantFunctionDescriptors :
SubtargetFeature<"invariant-function-descriptors",
"HasInvariantFunctionDescriptors", "true",
@@ -506,7 +516,8 @@ def ProcessorFeatures {
FeatureQuadwordAtomic,
FeaturePredictableSelectIsExpensive,
FeatureISA2_07,
- FeatureCRBits
+ FeatureCRBits,
+ FeatureFenceKeepsReservation
];
list<SubtargetFeature> P8SpecificFeatures = [FeatureAddiLoadFusion,
diff --git a/llvm/lib/Target/PowerPC/PPCISelLowering.cpp b/llvm/lib/Target/PowerPC/PPCISelLowering.cpp
index b012e6c6093ae5..bac60029fbcd5a 100644
--- a/llvm/lib/Target/PowerPC/PPCISelLowering.cpp
+++ b/llvm/lib/Target/PowerPC/PPCISelLowering.cpp
@@ -13320,6 +13320,10 @@ Instruction *PPCTargetLowering::emitTrailingFence(IRBuilderBase &Builder,
return nullptr;
}
+bool PPCTargetLowering::fenceClearsLoadLinkedReservation() const {
+ return !Subtarget.fenceKeepsReservation();
+}
+
MachineBasicBlock *PPCTargetLowering::EmitAtomicBinary(MachineInstr &MI,
MachineBasicBlock *BB,
unsigned BinOpcode,
diff --git a/llvm/lib/Target/PowerPC/PPCISelLowering.h b/llvm/lib/Target/PowerPC/PPCISelLowering.h
index 8086e56a796f79..64958275199cac 100644
--- a/llvm/lib/Target/PowerPC/PPCISelLowering.h
+++ b/llvm/lib/Target/PowerPC/PPCISelLowering.h
@@ -339,11 +339,11 @@ namespace llvm {
}
/// The Power ISA lets an implementation clear a reservation for reasons of
- /// its own, and e500v2 does so for a sync between the lwarx and the
- /// stwcx., which leaves a weak cmpxchg unable to ever succeed. Gating on
- /// isE500() would miss generic powerpc builds, where nothing enables it;
- /// the cost elsewhere is one fence on the comparison-failed path.
- bool fenceClearsLoadLinkedReservation() const override { return true; }
+ /// its own, and e500v2 loses it when a fence sits between the lwarx and
+ /// the stwcx., which leaves a weak cmpxchg unable to ever succeed. Answer
+ /// true unless the subtarget has FeatureFenceKeepsReservation, so CPUs
+ /// without it, including ppc and ppc64, get the conservative placement.
+ bool fenceClearsLoadLinkedReservation() const override;
Value *emitLoadLinked(IRBuilderBase &Builder, Type *ValueTy, Value *Addr,
AtomicOrdering Ord) const override;
diff --git a/llvm/test/CodeGen/PowerPC/atomics.ll b/llvm/test/CodeGen/PowerPC/atomics.ll
index 30897e06cbea1c..b718b259ead32c 100644
--- a/llvm/test/CodeGen/PowerPC/atomics.ll
+++ b/llvm/test/CodeGen/PowerPC/atomics.ll
@@ -1,7 +1,8 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
; RUN: llc -verify-machineinstrs < %s -mtriple=powerpc-unknown-linux-gnu -verify-machineinstrs -ppc-asm-full-reg-names | FileCheck %s --check-prefix=CHECK --check-prefix=PPC32
; This is already checked for in Atomics-64.ll
-; RUN: llc -verify-machineinstrs < %s -mcpu=ppc -mtriple=powerpc64-unknown-linux-gnu -ppc-asm-full-reg-names | FileCheck %s --check-prefix=CHECK --check-prefix=PPC64
+; RUN: llc -verify-machineinstrs < %s -mcpu=ppc -mtriple=powerpc64-unknown-linux-gnu -ppc-asm-full-reg-names | FileCheck %s --check-prefix=CHECK --check-prefix=PPC64 --check-prefix=PPC64-HOIST
+; RUN: llc -verify-machineinstrs < %s -mcpu=ppc -mattr=+fence-keeps-reservation -mtriple=powerpc64-unknown-linux-gnu -ppc-asm-full-reg-names | FileCheck %s --check-prefix=CHECK --check-prefix=PPC64 --check-prefix=PPC64-SINK
; FIXME: we don't currently check for the operations themselves with CHECK-NEXT,
; because they are implemented in a very messy way with lwarx/stwcx.
@@ -308,17 +309,29 @@ define i64 @cas_weak_i64_release_monotonic(ptr %mem) {
; PPC32-NEXT: mtlr r0
; PPC32-NEXT: blr
;
-; PPC64-LABEL: cas_weak_i64_release_monotonic:
-; PPC64: # %bb.0: # %cmpxchg.start
-; PPC64-NEXT: lwsync
-; PPC64-NEXT: mr r4, r3
-; PPC64-NEXT: ldarx r3, 0, r3
-; PPC64-NEXT: cmpldi r3, 0
-; PPC64-NEXT: bnelr- cr0
-; PPC64-NEXT: # %bb.1: # %cmpxchg.fencedstore
-; PPC64-NEXT: li r5, 1
-; PPC64-NEXT: stdcx. r5, 0, r4
-; PPC64-NEXT: blr
+; PPC64-HOIST-LABEL: cas_weak_i64_release_monotonic:
+; PPC64-HOIST: # %bb.0: # %cmpxchg.start
+; PPC64-HOIST-NEXT: lwsync
+; PPC64-HOIST-NEXT: mr r4, r3
+; PPC64-HOIST-NEXT: ldarx r3, 0, r3
+; PPC64-HOIST-NEXT: cmpldi r3, 0
+; PPC64-HOIST-NEXT: bnelr- cr0
+; PPC64-HOIST-NEXT: # %bb.1: # %cmpxchg.fencedstore
+; PPC64-HOIST-NEXT: li r5, 1
+; PPC64-HOIST-NEXT: stdcx. r5, 0, r4
+; PPC64-HOIST-NEXT: blr
+;
+; PPC64-SINK-LABEL: cas_weak_i64_release_monotonic:
+; PPC64-SINK: # %bb.0: # %cmpxchg.start
+; PPC64-SINK-NEXT: mr r4, r3
+; PPC64-SINK-NEXT: ldarx r3, 0, r3
+; PPC64-SINK-NEXT: cmpldi r3, 0
+; PPC64-SINK-NEXT: bnelr- cr0
+; PPC64-SINK-NEXT: # %bb.1: # %cmpxchg.fencedstore
+; PPC64-SINK-NEXT: lwsync
+; PPC64-SINK-NEXT: li r5, 1
+; PPC64-SINK-NEXT: stdcx. r5, 0, r4
+; PPC64-SINK-NEXT: blr
%val = cmpxchg weak ptr %mem, i64 0, i64 1 release monotonic
%loaded = extractvalue { i64, i1} %val, 0
ret i64 %loaded
diff --git a/llvm/test/Transforms/AtomicExpand/PowerPC/weak-cmpxchg-fence-placement.ll b/llvm/test/Transforms/AtomicExpand/PowerPC/weak-cmpxchg-fence-placement.ll
index 1a0fd06eb03a36..30d0c4bb0b0099 100644
--- a/llvm/test/Transforms/AtomicExpand/PowerPC/weak-cmpxchg-fence-placement.ll
+++ b/llvm/test/Transforms/AtomicExpand/PowerPC/weak-cmpxchg-fence-placement.ll
@@ -1,47 +1,93 @@
; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 5
; RUN: opt -S -mtriple=powerpc-unknown-linux-musl \
-; RUN: -passes='require<libcall-lowering-info>,atomic-expand' %s | FileCheck %s
+; RUN: -passes='require<libcall-lowering-info>,atomic-expand' %s \
+; RUN: | FileCheck %s --check-prefixes=CHECK,HOIST
+; RUN: opt -S -mtriple=powerpcspe-unknown-linux-musl \
+; RUN: -passes='require<libcall-lowering-info>,atomic-expand' %s \
+; RUN: | FileCheck %s --check-prefixes=CHECK,HOIST
+; RUN: opt -S -mtriple=powerpc64-unknown-linux-gnu \
+; RUN: -passes='require<libcall-lowering-info>,atomic-expand' %s \
+; RUN: | FileCheck %s --check-prefixes=CHECK,HOIST
+; RUN: opt -S -mtriple=powerpc64le-unknown-linux-gnu \
+; RUN: -passes='require<libcall-lowering-info>,atomic-expand' %s \
+; RUN: | FileCheck %s --check-prefixes=CHECK,SINK
-; The release barrier for a weak cmpxchg belongs before the load-linked, not
-; between it and the store-conditional. A barrier inside the reservation window
-; can clear the reservation, and a weak cmpxchg has no retry to recover with, so
-; sinking it there leaves an operation that can never succeed. The strong form
-; keeps the sunk barrier: its retry re-reserves afterwards, so it converges
-; either way.
+; A release barrier between the load-linked and the store-conditional can clear
+; the reservation on some implementations, and a weak cmpxchg has no retry to
+; recover with, so sinking it there can leave an operation that never succeeds.
+; The weak form therefore gets the barrier before the load-linked (HOIST),
+; unless the CPU has fence-keeps-reservation (SINK). Each RUN line uses the
+; default CPU of its triple: ppc, e500 and ppc64 lack the feature, while
+; ppc64le has the pwr8 features, fence-keeps-reservation among them. The
+; strong form keeps the sunk barrier everywhere: its retry re-reserves
+; afterwards, so it converges either way.
define i1 @weak_acq_rel(ptr %addr) {
-; CHECK-LABEL: define i1 @weak_acq_rel(
-; CHECK-SAME: ptr [[ADDR:%.*]]) {
-; CHECK-NEXT: call void @llvm.ppc.lwsync()
-; CHECK-NEXT: br label %[[CMPXCHG_START:.*]]
-; CHECK: [[CMPXCHG_START]]:
-; CHECK-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
-; CHECK-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
-; CHECK-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0:![0-9]+]]
-; CHECK: [[CMPXCHG_FENCEDSTORE]]:
-; CHECK-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
-; CHECK: [[CMPXCHG_TRYSTORE]]:
-; CHECK-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ]
-; CHECK-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
-; CHECK-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
-; CHECK-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
-; CHECK-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_FAILURE:.*]], !prof [[PROF0]]
-; CHECK: [[CMPXCHG_RELEASEDLOAD:.*:]]
-; CHECK-NEXT: unreachable
-; CHECK: [[CMPXCHG_SUCCESS]]:
-; CHECK-NEXT: call void @llvm.ppc.lwsync()
-; CHECK-NEXT: br label %[[CMPXCHG_END:.*]]
-; CHECK: [[CMPXCHG_NOSTORE]]:
-; CHECK-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ]
-; CHECK-NEXT: br label %[[CMPXCHG_FAILURE]]
-; CHECK: [[CMPXCHG_FAILURE]]:
-; CHECK-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ], [ [[LOADED_TRYSTORE]], %[[CMPXCHG_TRYSTORE]] ]
-; CHECK-NEXT: call void @llvm.ppc.lwsync()
-; CHECK-NEXT: br label %[[CMPXCHG_END]]
-; CHECK: [[CMPXCHG_END]]:
-; CHECK-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
-; CHECK-NEXT: [[SUCCESS1:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
-; CHECK-NEXT: ret i1 [[SUCCESS1]]
+; HOIST-LABEL: define i1 @weak_acq_rel(
+; HOIST-SAME: ptr [[ADDR:%.*]]) {
+; HOIST-NEXT: call void @llvm.ppc.lwsync()
+; HOIST-NEXT: br label %[[CMPXCHG_START:.*]]
+; HOIST: [[CMPXCHG_START]]:
+; HOIST-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
+; HOIST-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
+; HOIST-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0:![0-9]+]]
+; HOIST: [[CMPXCHG_FENCEDSTORE]]:
+; HOIST-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
+; HOIST: [[CMPXCHG_TRYSTORE]]:
+; HOIST-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ]
+; HOIST-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
+; HOIST-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
+; HOIST-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
+; HOIST-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_FAILURE:.*]], !prof [[PROF0]]
+; HOIST: [[CMPXCHG_RELEASEDLOAD:.*:]]
+; HOIST-NEXT: unreachable
+; HOIST: [[CMPXCHG_SUCCESS]]:
+; HOIST-NEXT: call void @llvm.ppc.lwsync()
+; HOIST-NEXT: br label %[[CMPXCHG_END:.*]]
+; HOIST: [[CMPXCHG_NOSTORE]]:
+; HOIST-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ]
+; HOIST-NEXT: br label %[[CMPXCHG_FAILURE]]
+; HOIST: [[CMPXCHG_FAILURE]]:
+; HOIST-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ], [ [[LOADED_TRYSTORE]], %[[CMPXCHG_TRYSTORE]] ]
+; HOIST-NEXT: call void @llvm.ppc.lwsync()
+; HOIST-NEXT: br label %[[CMPXCHG_END]]
+; HOIST: [[CMPXCHG_END]]:
+; HOIST-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
+; HOIST-NEXT: [[SUCCESS1:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
+; HOIST-NEXT: ret i1 [[SUCCESS1]]
+;
+; SINK-LABEL: define i1 @weak_acq_rel(
+; SINK-SAME: ptr [[ADDR:%.*]]) {
+; SINK-NEXT: br label %[[CMPXCHG_START:.*]]
+; SINK: [[CMPXCHG_START]]:
+; SINK-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
+; SINK-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
+; SINK-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0:![0-9]+]]
+; SINK: [[CMPXCHG_FENCEDSTORE]]:
+; SINK-NEXT: call void @llvm.ppc.lwsync()
+; SINK-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
+; SINK: [[CMPXCHG_TRYSTORE]]:
+; SINK-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ]
+; SINK-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
+; SINK-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
+; SINK-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
+; SINK-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_FAILURE:.*]], !prof [[PROF0]]
+; SINK: [[CMPXCHG_RELEASEDLOAD:.*:]]
+; SINK-NEXT: unreachable
+; SINK: [[CMPXCHG_SUCCESS]]:
+; SINK-NEXT: call void @llvm.ppc.lwsync()
+; SINK-NEXT: br label %[[CMPXCHG_END:.*]]
+; SINK: [[CMPXCHG_NOSTORE]]:
+; SINK-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ]
+; SINK-NEXT: br label %[[CMPXCHG_FAILURE]]
+; SINK: [[CMPXCHG_FAILURE]]:
+; SINK-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ], [ [[LOADED_TRYSTORE]], %[[CMPXCHG_TRYSTORE]] ]
+; SINK-NEXT: call void @llvm.ppc.lwsync()
+; SINK-NEXT: br label %[[CMPXCHG_END]]
+; SINK: [[CMPXCHG_END]]:
+; SINK-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
+; SINK-NEXT: [[SUCCESS1:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
+; SINK-NEXT: ret i1 [[SUCCESS1]]
;
%pair = cmpxchg weak ptr %addr, i32 0, i32 1 acq_rel acquire
%ok = extractvalue { i32, i1 } %pair, 1
@@ -49,42 +95,145 @@ define i1 @weak_acq_rel(ptr %addr) {
}
define i1 @weak_release(ptr %addr) {
-; CHECK-LABEL: define i1 @weak_release(
-; CHECK-SAME: ptr [[ADDR:%.*]]) {
-; CHECK-NEXT: call void @llvm.ppc.lwsync()
-; CHECK-NEXT: br label %[[CMPXCHG_START:.*]]
-; CHECK: [[CMPXCHG_START]]:
-; CHECK-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
-; CHECK-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
-; CHECK-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0]]
-; CHECK: [[CMPXCHG_FENCEDSTORE]]:
-; CHECK-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
-; CHECK: [[CMPXCHG_TRYSTORE]]:
-; CHECK-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ]
-; CHECK-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
-; CHECK-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
-; CHECK-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
-; CHECK-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_FAILURE:.*]], !prof [[PROF0]]
-; CHECK: [[CMPXCHG_RELEASEDLOAD:.*:]]
-; CHECK-NEXT: unreachable
-; CHECK: [[CMPXCHG_SUCCESS]]:
-; CHECK-NEXT: br label %[[CMPXCHG_END:.*]]
-; CHECK: [[CMPXCHG_NOSTORE]]:
-; CHECK-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ]
-; CHECK-NEXT: br label %[[CMPXCHG_FAILURE]]
-; CHECK: [[CMPXCHG_FAILURE]]:
-; CHECK-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ], [ [[LOADED_TRYSTORE]], %[[CMPXCHG_TRYSTORE]] ]
-; CHECK-NEXT: br label %[[CMPXCHG_END]]
-; CHECK: [[CMPXCHG_END]]:
-; CHECK-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
-; CHECK-NEXT: [[SUCCESS1:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
-; CHECK-NEXT: ret i1 [[SUCCESS1]]
+; HOIST-LABEL: define i1 @weak_release(
+; HOIST-SAME: ptr [[ADDR:%.*]]) {
+; HOIST-NEXT: call void @llvm.ppc.lwsync()
+; HOIST-NEXT: br label %[[CMPXCHG_START:.*]]
+; HOIST: [[CMPXCHG_START]]:
+; HOIST-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
+; HOIST-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
+; HOIST-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0]]
+; HOIST: [[CMPXCHG_FENCEDSTORE]]:
+; HOIST-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
+; HOIST: [[CMPXCHG_TRYSTORE]]:
+; HOIST-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ]
+; HOIST-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
+; HOIST-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
+; HOIST-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
+; HOIST-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_FAILURE:.*]], !prof [[PROF0]]
+; HOIST: [[CMPXCHG_RELEASEDLOAD:.*:]]
+; HOIST-NEXT: unreachable
+; HOIST: [[CMPXCHG_SUCCESS]]:
+; HOIST-NEXT: br label %[[CMPXCHG_END:.*]]
+; HOIST: [[CMPXCHG_NOSTORE]]:
+; HOIST-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ]
+; HOIST-NEXT: br label %[[CMPXCHG_FAILURE]]
+; HOIST: [[CMPXCHG_FAILURE]]:
+; HOIST-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ], [ [[LOADED_TRYSTORE]], %[[CMPXCHG_TRYSTORE]] ]
+; HOIST-NEXT: br label %[[CMPXCHG_END]]
+; HOIST: [[CMPXCHG_END]]:
+; HOIST-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
+; HOIST-NEXT: [[SUCCESS1:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
+; HOIST-NEXT: ret i1 [[SUCCESS1]]
+;
+; SINK-LABEL: define i1 @weak_release(
+; SINK-SAME: ptr [[ADDR:%.*]]) {
+; SINK-NEXT: br label %[[CMPXCHG_START:.*]]
+; SINK: [[CMPXCHG_START]]:
+; SINK-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
+; SINK-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
+; SINK-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0]]
+; SINK: [[CMPXCHG_FENCEDSTORE]]:
+; SINK-NEXT: call void @llvm.ppc.lwsync()
+; SINK-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
+; SINK: [[CMPXCHG_TRYSTORE]]:
+; SINK-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ]
+; SINK-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
+; SINK-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
+; SINK-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
+; SINK-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_FAILURE:.*]], !prof [[PROF0]]
+; SINK: [[CMPXCHG_RELEASEDLOAD:.*:]]
+; SINK-NEXT: unreachable
+; SINK: [[CMPXCHG_SUCCESS]]:
+; SINK-NEXT: br label %[[CMPXCHG_END:.*]]
+; SINK: [[CMPXCHG_NOSTORE]]:
+; SINK-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ]
+; SINK-NEXT: br label %[[CMPXCHG_FAILURE]]
+; SINK: [[CMPXCHG_FAILURE]]:
+; SINK-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ], [ [[LOADED_TRYSTORE]], %[[CMPXCHG_TRYSTORE]] ]
+; SINK-NEXT: br label %[[CMPXCHG_END]]
+; SINK: [[CMPXCHG_END]]:
+; SINK-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
+; SINK-NEXT: [[SUCCESS1:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
+; SINK-NEXT: ret i1 [[SUCCESS1]]
;
%pair = cmpxchg weak ptr %addr, i32 0, i32 1 release monotonic
%ok = extractvalue { i32, i1 } %pair, 1
ret i1 %ok
}
+define i1 @weak_seq_cst(ptr %addr) {
+; HOIST-LABEL: define i1 @weak_seq_cst(
+; HOIST-SAME: ptr [[ADDR:%.*]]) {
+; HOIST-NEXT: call void @llvm.ppc.sync()
+; HOIST-NEXT: br label %[[CMPXCHG_START:.*]]
+; HOIST: [[CMPXCHG_START]]:
+; HOIST-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
+; HOIST-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
+; HOIST-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0]]
+; HOIST: [[CMPXCHG_FENCEDSTORE]]:
+; HOIST-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
+; HOIST: [[CMPXCHG_TRYSTORE]]:
+; HOIST-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ]
+; HOIST-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
+; HOIST-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
+; HOIST-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
+; HOIST-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_FAILURE:.*]], !prof [[PROF0]]
+; HOIST: [[CMPXCHG_RELEASEDLOAD:.*:]]
+; HOIST-NEXT: unreachable
+; HOIST: [[CMPXCHG_SUCCESS]]:
+; HOIST-NEXT: call void @llvm.ppc.lwsync()
+; HOIST-NEXT: br label %[[CMPXCHG_END:.*]]
+; HOIST: [[CMPXCHG_NOSTORE]]:
+; HOIST-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ]
+; HOIST-NEXT: br label %[[CMPXCHG_FAILURE]]
+; HOIST: [[CMPXCHG_FAILURE]]:
+; HOIST-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ], [ [[LOADED_TRYSTORE]], %[[CMPXCHG_TRYSTORE]] ]
+; HOIST-NEXT: call void @llvm.ppc.lwsync()
+; HOIST-NEXT: br label %[[CMPXCHG_END]]
+; HOIST: [[CMPXCHG_END]]:
+; HOIST-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
+; HOIST-NEXT: [[SUCCESS1:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
+; HOIST-NEXT: ret i1 [[SUCCESS1]]
+;
+; SINK-LABEL: define i1 @weak_seq_cst(
+; SINK-SAME: ptr [[ADDR:%.*]]) {
+; SINK-NEXT: br label %[[CMPXCHG_START:.*]]
+; SINK: [[CMPXCHG_START]]:
+; SINK-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
+; SINK-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
+; SINK-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0]]
+; SINK: [[CMPXCHG_FENCEDSTORE]]:
+; SINK-NEXT: call void @llvm.ppc.sync()
+; SINK-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
+; SINK: [[CMPXCHG_TRYSTORE]]:
+; SINK-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ]
+; SINK-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
+; SINK-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
+; SINK-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
+; SINK-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_FAILURE:.*]], !prof [[PROF0]]
+; SINK: [[CMPXCHG_RELEASEDLOAD:.*:]]
+; SINK-NEXT: unreachable
+; SINK: [[CMPXCHG_SUCCESS]]:
+; SINK-NEXT: call void @llvm.ppc.lwsync()
+; SINK-NEXT: br label %[[CMPXCHG_END:.*]]
+; SINK: [[CMPXCHG_NOSTORE]]:
+; SINK-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ]
+; SINK-NEXT: br label %[[CMPXCHG_FAILURE]]
+; SINK: [[CMPXCHG_FAILURE]]:
+; SINK-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ], [ [[LOADED_TRYSTORE]], %[[CMPXCHG_TRYSTORE]] ]
+; SINK-NEXT: call void @llvm.ppc.lwsync()
+; SINK-NEXT: br label %[[CMPXCHG_END]]
+; SINK: [[CMPXCHG_END]]:
+; SINK-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
+; SINK-NEXT: [[SUCCESS1:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
+; SINK-NEXT: ret i1 [[SUCCESS1]]
+;
+ %pair = cmpxchg weak ptr %addr, i32 0, i32 1 seq_cst seq_cst
+ %ok = extractvalue { i32, i1 } %pair, 1
+ ret i1 %ok
+}
+
define i1 @strong_acq_rel(ptr %addr) {
; CHECK-LABEL: define i1 @strong_acq_rel(
; CHECK-SAME: ptr [[ADDR:%.*]]) {
@@ -92,7 +241,7 @@ define i1 @strong_acq_rel(ptr %addr) {
; CHECK: [[CMPXCHG_START]]:
; CHECK-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
; CHECK-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
-; CHECK-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0]]
+; CHECK-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0:![0-9]+]]
; CHECK: [[CMPXCHG_FENCEDSTORE]]:
; CHECK-NEXT: call void @llvm.ppc.lwsync()
; CHECK-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
@@ -162,5 +311,7 @@ define i1 @weak_monotonic(ptr %addr) {
ret i1 %ok
}
;.
-; CHECK: [[PROF0]] = !{!"branch_weights", i32 1048575, i32 1}
+; HOIST: [[PROF0]] = !{!"branch_weights", i32 1048575, i32 1}
+;.
+; SINK: [[PROF0]] = !{!"branch_weights", i32 1048575, i32 1}
;.
>From ec197a638526dd12caa5313bd0be15f34eabb545 Mon Sep 17 00:00:00 2001
From: Josef Schlehofer <pepe.schlehofer at gmail.com>
Date: Wed, 23 Sep 2026 14:44:56 +0200
Subject: [PATCH 3/3] fixup! [AtomicExpand] Let targets keep the release fence
out of the reservation
Shorten the comments and reduce the AtomicExpand test to the fence placement
checks. The test now also covers e500 and +fence-keeps-reservation.
---
llvm/include/llvm/CodeGen/TargetLowering.h | 8 +-
llvm/lib/CodeGen/AtomicExpandPass.cpp | 10 +-
llvm/lib/Target/PowerPC/PPC.td | 11 +-
llvm/lib/Target/PowerPC/PPCISelLowering.h | 6 +-
.../PowerPC/weak-cmpxchg-fence-placement.ll | 323 ++----------------
5 files changed, 40 insertions(+), 318 deletions(-)
diff --git a/llvm/include/llvm/CodeGen/TargetLowering.h b/llvm/include/llvm/CodeGen/TargetLowering.h
index ff4797ce768308..1c427bc894f3d1 100644
--- a/llvm/include/llvm/CodeGen/TargetLowering.h
+++ b/llvm/include/llvm/CodeGen/TargetLowering.h
@@ -2313,11 +2313,9 @@ class LLVM_ABI TargetLoweringBase {
return false;
}
- /// Whether a fence placed between the load-linked and the store-conditional
- /// can clear the reservation on this target. When it can, AtomicExpandPass
- /// must not sink the leading fence of a weak cmpxchg into the reservation
- /// window: the store-conditional would fail, and a weak cmpxchg has no retry
- /// to re-reserve and recover with. Defaults to false.
+ /// Whether a fence between the load-linked and the store-conditional can
+ /// clear the reservation. If true, AtomicExpandPass places the release fence
+ /// of a weak cmpxchg before the load-linked. Defaults to false.
virtual bool fenceClearsLoadLinkedReservation() const { return false; }
/// Whether AtomicExpandPass should automatically insert a seq_cst trailing
diff --git a/llvm/lib/CodeGen/AtomicExpandPass.cpp b/llvm/lib/CodeGen/AtomicExpandPass.cpp
index 707800e96e1f20..c9ceaa8ee6082d 100644
--- a/llvm/lib/CodeGen/AtomicExpandPass.cpp
+++ b/llvm/lib/CodeGen/AtomicExpandPass.cpp
@@ -1498,14 +1498,8 @@ bool AtomicExpandImpl::expandAtomicCmpXchg(AtomicCmpXchgInst *CI) {
// There's no overhead for sinking the release barrier in a weak cmpxchg, so
// do it even on minsize.
- //
- // Except where a fence can clear the reservation. Sinking puts the fence
- // between the load-linked and the store-conditional, and there the
- // store-conditional fails; the strong form recovers because its retry
- // re-reserves afterwards, but a weak cmpxchg has no retry and so could never
- // succeed. Hoisting is valid because the fence is only needed to provide the
- // release ordering of a successful store, and executing it on an attempt
- // that fails before the store has no additional ordering effect.
+ // A fence between the LL and SC may clear the reservation on some targets.
+ // Strong cmpxchg retries, but weak cmpxchg cannot recover.
bool UseUnconditionalReleaseBarrier =
(F->hasMinSize() && !CI->isWeak()) ||
(CI->isWeak() && TLI->fenceClearsLoadLinkedReservation());
diff --git a/llvm/lib/Target/PowerPC/PPC.td b/llvm/lib/Target/PowerPC/PPC.td
index 312380297d80c8..19d29ca70b5404 100644
--- a/llvm/lib/Target/PowerPC/PPC.td
+++ b/llvm/lib/Target/PowerPC/PPC.td
@@ -177,16 +177,11 @@ def FeaturePartwordAtomic : SubtargetFeature<"partword-atomics",
def FeatureQuadwordAtomic : SubtargetFeature<"quadword-atomics",
"HasQuadwordAtomics", "true",
"Enable lqarx and stqcx.">;
-// The Power ISA lets an implementation clear a reservation for reasons of its
-// own, and some clear it when a sync or lwsync is executed between the larx
-// and the stcx. That leaves a weak cmpxchg whose release fence is placed there
-// unable to succeed, so AtomicExpand only places it there on processors with
-// this feature. All others, including ppc, ppc32 and ppc64, get the fence
-// ahead of the larx.
+// Some PowerPC implementations clear the LL/SC reservation on a fence.
def FeatureFenceKeepsReservation :
SubtargetFeature<"fence-keeps-reservation", "FenceKeepsReservation", "true",
- "A sync or lwsync between larx and stcx. does not clear "
- "the reservation", [], InlineIgnore>;
+ "A fence between larx and stcx. keeps the reservation", [],
+ InlineIgnore>;
def FeatureInvariantFunctionDescriptors :
SubtargetFeature<"invariant-function-descriptors",
"HasInvariantFunctionDescriptors", "true",
diff --git a/llvm/lib/Target/PowerPC/PPCISelLowering.h b/llvm/lib/Target/PowerPC/PPCISelLowering.h
index 64958275199cac..1874a4660b02d8 100644
--- a/llvm/lib/Target/PowerPC/PPCISelLowering.h
+++ b/llvm/lib/Target/PowerPC/PPCISelLowering.h
@@ -338,11 +338,7 @@ namespace llvm {
return true;
}
- /// The Power ISA lets an implementation clear a reservation for reasons of
- /// its own, and e500v2 loses it when a fence sits between the lwarx and
- /// the stwcx., which leaves a weak cmpxchg unable to ever succeed. Answer
- /// true unless the subtarget has FeatureFenceKeepsReservation, so CPUs
- /// without it, including ppc and ppc64, get the conservative placement.
+ /// True unless the subtarget has FeatureFenceKeepsReservation.
bool fenceClearsLoadLinkedReservation() const override;
Value *emitLoadLinked(IRBuilderBase &Builder, Type *ValueTy, Value *Addr,
diff --git a/llvm/test/Transforms/AtomicExpand/PowerPC/weak-cmpxchg-fence-placement.ll b/llvm/test/Transforms/AtomicExpand/PowerPC/weak-cmpxchg-fence-placement.ll
index 30d0c4bb0b0099..bf405f28f99b5a 100644
--- a/llvm/test/Transforms/AtomicExpand/PowerPC/weak-cmpxchg-fence-placement.ll
+++ b/llvm/test/Transforms/AtomicExpand/PowerPC/weak-cmpxchg-fence-placement.ll
@@ -1,317 +1,56 @@
-; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 5
; RUN: opt -S -mtriple=powerpc-unknown-linux-musl \
; RUN: -passes='require<libcall-lowering-info>,atomic-expand' %s \
; RUN: | FileCheck %s --check-prefixes=CHECK,HOIST
-; RUN: opt -S -mtriple=powerpcspe-unknown-linux-musl \
+; RUN: opt -S -mtriple=powerpc-unknown-linux-musl -mcpu=e500 \
; RUN: -passes='require<libcall-lowering-info>,atomic-expand' %s \
; RUN: | FileCheck %s --check-prefixes=CHECK,HOIST
-; RUN: opt -S -mtriple=powerpc64-unknown-linux-gnu \
+; RUN: opt -S -mtriple=powerpc-unknown-linux-musl \
+; RUN: -mattr=+fence-keeps-reservation \
; RUN: -passes='require<libcall-lowering-info>,atomic-expand' %s \
-; RUN: | FileCheck %s --check-prefixes=CHECK,HOIST
+; RUN: | FileCheck %s --check-prefixes=CHECK,SINK
; RUN: opt -S -mtriple=powerpc64le-unknown-linux-gnu \
; RUN: -passes='require<libcall-lowering-info>,atomic-expand' %s \
; RUN: | FileCheck %s --check-prefixes=CHECK,SINK
-; A release barrier between the load-linked and the store-conditional can clear
-; the reservation on some implementations, and a weak cmpxchg has no retry to
-; recover with, so sinking it there can leave an operation that never succeeds.
-; The weak form therefore gets the barrier before the load-linked (HOIST),
-; unless the CPU has fence-keeps-reservation (SINK). Each RUN line uses the
-; default CPU of its triple: ppc, e500 and ppc64 lack the feature, while
-; ppc64le has the pwr8 features, fence-keeps-reservation among them. The
-; strong form keeps the sunk barrier everywhere: its retry re-reserves
-; afterwards, so it converges either way.
-
-define i1 @weak_acq_rel(ptr %addr) {
-; HOIST-LABEL: define i1 @weak_acq_rel(
-; HOIST-SAME: ptr [[ADDR:%.*]]) {
-; HOIST-NEXT: call void @llvm.ppc.lwsync()
-; HOIST-NEXT: br label %[[CMPXCHG_START:.*]]
-; HOIST: [[CMPXCHG_START]]:
-; HOIST-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
-; HOIST-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
-; HOIST-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0:![0-9]+]]
-; HOIST: [[CMPXCHG_FENCEDSTORE]]:
-; HOIST-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
-; HOIST: [[CMPXCHG_TRYSTORE]]:
-; HOIST-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ]
-; HOIST-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
-; HOIST-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
-; HOIST-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
-; HOIST-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_FAILURE:.*]], !prof [[PROF0]]
-; HOIST: [[CMPXCHG_RELEASEDLOAD:.*:]]
-; HOIST-NEXT: unreachable
-; HOIST: [[CMPXCHG_SUCCESS]]:
-; HOIST-NEXT: call void @llvm.ppc.lwsync()
-; HOIST-NEXT: br label %[[CMPXCHG_END:.*]]
-; HOIST: [[CMPXCHG_NOSTORE]]:
-; HOIST-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ]
-; HOIST-NEXT: br label %[[CMPXCHG_FAILURE]]
-; HOIST: [[CMPXCHG_FAILURE]]:
-; HOIST-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ], [ [[LOADED_TRYSTORE]], %[[CMPXCHG_TRYSTORE]] ]
-; HOIST-NEXT: call void @llvm.ppc.lwsync()
-; HOIST-NEXT: br label %[[CMPXCHG_END]]
-; HOIST: [[CMPXCHG_END]]:
-; HOIST-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
-; HOIST-NEXT: [[SUCCESS1:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
-; HOIST-NEXT: ret i1 [[SUCCESS1]]
-;
-; SINK-LABEL: define i1 @weak_acq_rel(
-; SINK-SAME: ptr [[ADDR:%.*]]) {
-; SINK-NEXT: br label %[[CMPXCHG_START:.*]]
-; SINK: [[CMPXCHG_START]]:
-; SINK-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
-; SINK-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
-; SINK-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0:![0-9]+]]
-; SINK: [[CMPXCHG_FENCEDSTORE]]:
-; SINK-NEXT: call void @llvm.ppc.lwsync()
-; SINK-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
-; SINK: [[CMPXCHG_TRYSTORE]]:
-; SINK-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ]
-; SINK-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
-; SINK-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
-; SINK-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
-; SINK-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_FAILURE:.*]], !prof [[PROF0]]
-; SINK: [[CMPXCHG_RELEASEDLOAD:.*:]]
-; SINK-NEXT: unreachable
-; SINK: [[CMPXCHG_SUCCESS]]:
-; SINK-NEXT: call void @llvm.ppc.lwsync()
-; SINK-NEXT: br label %[[CMPXCHG_END:.*]]
-; SINK: [[CMPXCHG_NOSTORE]]:
-; SINK-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ]
-; SINK-NEXT: br label %[[CMPXCHG_FAILURE]]
-; SINK: [[CMPXCHG_FAILURE]]:
-; SINK-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ], [ [[LOADED_TRYSTORE]], %[[CMPXCHG_TRYSTORE]] ]
-; SINK-NEXT: call void @llvm.ppc.lwsync()
-; SINK-NEXT: br label %[[CMPXCHG_END]]
-; SINK: [[CMPXCHG_END]]:
-; SINK-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
-; SINK-NEXT: [[SUCCESS1:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
-; SINK-NEXT: ret i1 [[SUCCESS1]]
-;
- %pair = cmpxchg weak ptr %addr, i32 0, i32 1 acq_rel acquire
- %ok = extractvalue { i32, i1 } %pair, 1
- ret i1 %ok
-}
+; A weak cmpxchg gets its release fence before the lwarx (HOIST) unless the CPU
+; has fence-keeps-reservation (SINK). A strong cmpxchg always sinks it.
-define i1 @weak_release(ptr %addr) {
-; HOIST-LABEL: define i1 @weak_release(
-; HOIST-SAME: ptr [[ADDR:%.*]]) {
+define i1 @weak_release(ptr %p) {
+; CHECK-LABEL: define i1 @weak_release(
; HOIST-NEXT: call void @llvm.ppc.lwsync()
-; HOIST-NEXT: br label %[[CMPXCHG_START:.*]]
-; HOIST: [[CMPXCHG_START]]:
-; HOIST-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
-; HOIST-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
-; HOIST-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0]]
-; HOIST: [[CMPXCHG_FENCEDSTORE]]:
-; HOIST-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
-; HOIST: [[CMPXCHG_TRYSTORE]]:
-; HOIST-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ]
-; HOIST-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
-; HOIST-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
-; HOIST-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
-; HOIST-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_FAILURE:.*]], !prof [[PROF0]]
-; HOIST: [[CMPXCHG_RELEASEDLOAD:.*:]]
-; HOIST-NEXT: unreachable
-; HOIST: [[CMPXCHG_SUCCESS]]:
-; HOIST-NEXT: br label %[[CMPXCHG_END:.*]]
-; HOIST: [[CMPXCHG_NOSTORE]]:
-; HOIST-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ]
-; HOIST-NEXT: br label %[[CMPXCHG_FAILURE]]
-; HOIST: [[CMPXCHG_FAILURE]]:
-; HOIST-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ], [ [[LOADED_TRYSTORE]], %[[CMPXCHG_TRYSTORE]] ]
-; HOIST-NEXT: br label %[[CMPXCHG_END]]
-; HOIST: [[CMPXCHG_END]]:
-; HOIST-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
-; HOIST-NEXT: [[SUCCESS1:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
-; HOIST-NEXT: ret i1 [[SUCCESS1]]
-;
-; SINK-LABEL: define i1 @weak_release(
-; SINK-SAME: ptr [[ADDR:%.*]]) {
-; SINK-NEXT: br label %[[CMPXCHG_START:.*]]
-; SINK: [[CMPXCHG_START]]:
-; SINK-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
-; SINK-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
-; SINK-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0]]
-; SINK: [[CMPXCHG_FENCEDSTORE]]:
-; SINK-NEXT: call void @llvm.ppc.lwsync()
-; SINK-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
-; SINK: [[CMPXCHG_TRYSTORE]]:
-; SINK-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ]
-; SINK-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
-; SINK-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
-; SINK-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
-; SINK-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_FAILURE:.*]], !prof [[PROF0]]
-; SINK: [[CMPXCHG_RELEASEDLOAD:.*:]]
-; SINK-NEXT: unreachable
-; SINK: [[CMPXCHG_SUCCESS]]:
-; SINK-NEXT: br label %[[CMPXCHG_END:.*]]
-; SINK: [[CMPXCHG_NOSTORE]]:
-; SINK-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ]
-; SINK-NEXT: br label %[[CMPXCHG_FAILURE]]
-; SINK: [[CMPXCHG_FAILURE]]:
-; SINK-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ], [ [[LOADED_TRYSTORE]], %[[CMPXCHG_TRYSTORE]] ]
-; SINK-NEXT: br label %[[CMPXCHG_END]]
-; SINK: [[CMPXCHG_END]]:
-; SINK-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
-; SINK-NEXT: [[SUCCESS1:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
-; SINK-NEXT: ret i1 [[SUCCESS1]]
-;
- %pair = cmpxchg weak ptr %addr, i32 0, i32 1 release monotonic
+; SINK-NOT: @llvm.ppc.{{(lw)?}}sync
+; CHECK: call i32 @llvm.ppc.lwarx(ptr %p)
+; HOIST-NOT: @llvm.ppc.{{(lw)?}}sync
+; SINK: cmpxchg.fencedstore:
+; SINK-NEXT: call void @llvm.ppc.lwsync()
+; CHECK: call i32 @llvm.ppc.stwcx(ptr %p, i32 1)
+ %pair = cmpxchg weak ptr %p, i32 0, i32 1 release monotonic
%ok = extractvalue { i32, i1 } %pair, 1
ret i1 %ok
}
-define i1 @weak_seq_cst(ptr %addr) {
-; HOIST-LABEL: define i1 @weak_seq_cst(
-; HOIST-SAME: ptr [[ADDR:%.*]]) {
+define i1 @weak_seq_cst(ptr %p) {
+; CHECK-LABEL: define i1 @weak_seq_cst(
; HOIST-NEXT: call void @llvm.ppc.sync()
-; HOIST-NEXT: br label %[[CMPXCHG_START:.*]]
-; HOIST: [[CMPXCHG_START]]:
-; HOIST-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
-; HOIST-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
-; HOIST-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0]]
-; HOIST: [[CMPXCHG_FENCEDSTORE]]:
-; HOIST-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
-; HOIST: [[CMPXCHG_TRYSTORE]]:
-; HOIST-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ]
-; HOIST-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
-; HOIST-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
-; HOIST-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
-; HOIST-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_FAILURE:.*]], !prof [[PROF0]]
-; HOIST: [[CMPXCHG_RELEASEDLOAD:.*:]]
-; HOIST-NEXT: unreachable
-; HOIST: [[CMPXCHG_SUCCESS]]:
-; HOIST-NEXT: call void @llvm.ppc.lwsync()
-; HOIST-NEXT: br label %[[CMPXCHG_END:.*]]
-; HOIST: [[CMPXCHG_NOSTORE]]:
-; HOIST-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ]
-; HOIST-NEXT: br label %[[CMPXCHG_FAILURE]]
-; HOIST: [[CMPXCHG_FAILURE]]:
-; HOIST-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ], [ [[LOADED_TRYSTORE]], %[[CMPXCHG_TRYSTORE]] ]
-; HOIST-NEXT: call void @llvm.ppc.lwsync()
-; HOIST-NEXT: br label %[[CMPXCHG_END]]
-; HOIST: [[CMPXCHG_END]]:
-; HOIST-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
-; HOIST-NEXT: [[SUCCESS1:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
-; HOIST-NEXT: ret i1 [[SUCCESS1]]
-;
-; SINK-LABEL: define i1 @weak_seq_cst(
-; SINK-SAME: ptr [[ADDR:%.*]]) {
-; SINK-NEXT: br label %[[CMPXCHG_START:.*]]
-; SINK: [[CMPXCHG_START]]:
-; SINK-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
-; SINK-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
-; SINK-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0]]
-; SINK: [[CMPXCHG_FENCEDSTORE]]:
-; SINK-NEXT: call void @llvm.ppc.sync()
-; SINK-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
-; SINK: [[CMPXCHG_TRYSTORE]]:
-; SINK-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ]
-; SINK-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
-; SINK-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
-; SINK-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
-; SINK-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_FAILURE:.*]], !prof [[PROF0]]
-; SINK: [[CMPXCHG_RELEASEDLOAD:.*:]]
-; SINK-NEXT: unreachable
-; SINK: [[CMPXCHG_SUCCESS]]:
-; SINK-NEXT: call void @llvm.ppc.lwsync()
-; SINK-NEXT: br label %[[CMPXCHG_END:.*]]
-; SINK: [[CMPXCHG_NOSTORE]]:
-; SINK-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ]
-; SINK-NEXT: br label %[[CMPXCHG_FAILURE]]
-; SINK: [[CMPXCHG_FAILURE]]:
-; SINK-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ], [ [[LOADED_TRYSTORE]], %[[CMPXCHG_TRYSTORE]] ]
-; SINK-NEXT: call void @llvm.ppc.lwsync()
-; SINK-NEXT: br label %[[CMPXCHG_END]]
-; SINK: [[CMPXCHG_END]]:
-; SINK-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
-; SINK-NEXT: [[SUCCESS1:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
-; SINK-NEXT: ret i1 [[SUCCESS1]]
-;
- %pair = cmpxchg weak ptr %addr, i32 0, i32 1 seq_cst seq_cst
+; SINK-NOT: @llvm.ppc.{{(lw)?}}sync
+; CHECK: call i32 @llvm.ppc.lwarx(ptr %p)
+; HOIST-NOT: @llvm.ppc.{{(lw)?}}sync
+; SINK: cmpxchg.fencedstore:
+; SINK-NEXT: call void @llvm.ppc.sync()
+; CHECK: call i32 @llvm.ppc.stwcx(ptr %p, i32 1)
+ %pair = cmpxchg weak ptr %p, i32 0, i32 1 seq_cst seq_cst
%ok = extractvalue { i32, i1 } %pair, 1
ret i1 %ok
}
-define i1 @strong_acq_rel(ptr %addr) {
-; CHECK-LABEL: define i1 @strong_acq_rel(
-; CHECK-SAME: ptr [[ADDR:%.*]]) {
-; CHECK-NEXT: br label %[[CMPXCHG_START:.*]]
-; CHECK: [[CMPXCHG_START]]:
-; CHECK-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
-; CHECK-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
-; CHECK-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0:![0-9]+]]
-; CHECK: [[CMPXCHG_FENCEDSTORE]]:
-; CHECK-NEXT: call void @llvm.ppc.lwsync()
-; CHECK-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
-; CHECK: [[CMPXCHG_TRYSTORE]]:
-; CHECK-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ], [ [[LARX1:%.*]], %[[CMPXCHG_RELEASEDLOAD:.*]] ]
-; CHECK-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
-; CHECK-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
-; CHECK-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
-; CHECK-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_RELEASEDLOAD]], !prof [[PROF0]]
-; CHECK: [[CMPXCHG_RELEASEDLOAD]]:
-; CHECK-NEXT: [[LARX1]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
-; CHECK-NEXT: [[SHOULD_STORE2:%.*]] = icmp eq i32 [[LARX1]], 0
-; CHECK-NEXT: br i1 [[SHOULD_STORE2]], label %[[CMPXCHG_TRYSTORE]], label %[[CMPXCHG_NOSTORE]], !prof [[PROF0]]
-; CHECK: [[CMPXCHG_SUCCESS]]:
-; CHECK-NEXT: call void @llvm.ppc.lwsync()
-; CHECK-NEXT: br label %[[CMPXCHG_END:.*]]
-; CHECK: [[CMPXCHG_NOSTORE]]:
-; CHECK-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ], [ [[LARX1]], %[[CMPXCHG_RELEASEDLOAD]] ]
-; CHECK-NEXT: br label %[[CMPXCHG_FAILURE:.*]]
-; CHECK: [[CMPXCHG_FAILURE]]:
-; CHECK-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ]
+define i1 @strong_release(ptr %p) {
+; CHECK-LABEL: define i1 @strong_release(
+; CHECK-NOT: @llvm.ppc.{{(lw)?}}sync
+; CHECK: call i32 @llvm.ppc.lwarx(ptr %p)
+; CHECK: cmpxchg.fencedstore:
; CHECK-NEXT: call void @llvm.ppc.lwsync()
-; CHECK-NEXT: br label %[[CMPXCHG_END]]
-; CHECK: [[CMPXCHG_END]]:
-; CHECK-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
-; CHECK-NEXT: [[SUCCESS3:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
-; CHECK-NEXT: ret i1 [[SUCCESS3]]
-;
- %pair = cmpxchg ptr %addr, i32 0, i32 1 acq_rel acquire
- %ok = extractvalue { i32, i1 } %pair, 1
- ret i1 %ok
-}
-
-define i1 @weak_monotonic(ptr %addr) {
-; CHECK-LABEL: define i1 @weak_monotonic(
-; CHECK-SAME: ptr [[ADDR:%.*]]) {
-; CHECK-NEXT: br label %[[CMPXCHG_START:.*]]
-; CHECK: [[CMPXCHG_START]]:
-; CHECK-NEXT: [[LARX:%.*]] = call i32 @llvm.ppc.lwarx(ptr [[ADDR]])
-; CHECK-NEXT: [[SHOULD_STORE:%.*]] = icmp eq i32 [[LARX]], 0
-; CHECK-NEXT: br i1 [[SHOULD_STORE]], label %[[CMPXCHG_FENCEDSTORE:.*]], label %[[CMPXCHG_NOSTORE:.*]], !prof [[PROF0]]
-; CHECK: [[CMPXCHG_FENCEDSTORE]]:
-; CHECK-NEXT: br label %[[CMPXCHG_TRYSTORE:.*]]
-; CHECK: [[CMPXCHG_TRYSTORE]]:
-; CHECK-NEXT: [[LOADED_TRYSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_FENCEDSTORE]] ]
-; CHECK-NEXT: [[STCX:%.*]] = call i32 @llvm.ppc.stwcx(ptr [[ADDR]], i32 1)
-; CHECK-NEXT: [[TMP1:%.*]] = xor i32 [[STCX]], 1
-; CHECK-NEXT: [[SUCCESS:%.*]] = icmp eq i32 [[TMP1]], 0
-; CHECK-NEXT: br i1 [[SUCCESS]], label %[[CMPXCHG_SUCCESS:.*]], label %[[CMPXCHG_FAILURE:.*]], !prof [[PROF0]]
-; CHECK: [[CMPXCHG_RELEASEDLOAD:.*:]]
-; CHECK-NEXT: unreachable
-; CHECK: [[CMPXCHG_SUCCESS]]:
-; CHECK-NEXT: br label %[[CMPXCHG_END:.*]]
-; CHECK: [[CMPXCHG_NOSTORE]]:
-; CHECK-NEXT: [[LOADED_NOSTORE:%.*]] = phi i32 [ [[LARX]], %[[CMPXCHG_START]] ]
-; CHECK-NEXT: br label %[[CMPXCHG_FAILURE]]
-; CHECK: [[CMPXCHG_FAILURE]]:
-; CHECK-NEXT: [[LOADED_FAILURE:%.*]] = phi i32 [ [[LOADED_NOSTORE]], %[[CMPXCHG_NOSTORE]] ], [ [[LOADED_TRYSTORE]], %[[CMPXCHG_TRYSTORE]] ]
-; CHECK-NEXT: br label %[[CMPXCHG_END]]
-; CHECK: [[CMPXCHG_END]]:
-; CHECK-NEXT: [[LOADED_EXIT:%.*]] = phi i32 [ [[LOADED_TRYSTORE]], %[[CMPXCHG_SUCCESS]] ], [ [[LOADED_FAILURE]], %[[CMPXCHG_FAILURE]] ]
-; CHECK-NEXT: [[SUCCESS1:%.*]] = phi i1 [ true, %[[CMPXCHG_SUCCESS]] ], [ false, %[[CMPXCHG_FAILURE]] ]
-; CHECK-NEXT: ret i1 [[SUCCESS1]]
-;
- %pair = cmpxchg weak ptr %addr, i32 0, i32 1 monotonic monotonic
+; CHECK: call i32 @llvm.ppc.stwcx(ptr %p, i32 1)
+ %pair = cmpxchg ptr %p, i32 0, i32 1 release monotonic
%ok = extractvalue { i32, i1 } %pair, 1
ret i1 %ok
}
-;.
-; HOIST: [[PROF0]] = !{!"branch_weights", i32 1048575, i32 1}
-;.
-; SINK: [[PROF0]] = !{!"branch_weights", i32 1048575, i32 1}
-;.
More information about the flang-commits
mailing list