[clang] [llvm] [openmp] [OpenMP] Honour memory-order clauses on the flush directive (PR #241295)
via cfe-commits
cfe-commits at lists.llvm.org
Wed Nov 11 03:47:01 PST 2026
llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-clang-codegen
Author: Julian Brown (jtb20)
<details>
<summary>Changes</summary>
At present the acquire, release and acq_rel clauses on 'flush' are accepted but have no effect. Flushes are emitted as __kmpc_flush runtime calls, which implement a seq_cst fence.
AMD have had a downstream patch for a while (by Carlo Bertolli) which adds per-ordering __kmpc_flush_* entry points to the device runtime. This patch takes a slightly different approach: for the host, a seq_cst flush still calls __kmpc_flush, and a flush with any other ordering calls the __kmpc_flush_explicit entry point from #<!-- -->133325, so it needs a runtime that has that. For the GPU, it emits a fence directly.
This has mild benefits: OpenMPOpt already handles fences when it converts kernels to SPMD mode, but has no knowledge of the new __kmpc_flush calls (either _explicit or Carlo's per-ordering flush calls). Scope names differ per-GPU: they are determined using the mechanism introduced by
The lowering is done in OpenMPIRBuilder. The createFlush function now takes an ordering and a scope, and is now used for every GPU flush. The OpenMPIRBuilder-disabled path is unaffected. Flang/MLIR call createFlush with the defaults, so their host output is unchanged, but their GPU flushes, including those implied by atomics, now become fences instead of __kmpc_flush calls.
Since e8e05de08bea ("[OPENMP50]Add codegen for acq_rel clause in atomic|flush directives"), a flush with neither a list nor a memory-order clause has been passed to emitFlush as acq_rel. From OpenMP 5.1 it should behave as seq_cst instead, so it is now emitted as that.
Flushes implied by 'atomic' are also emitted as seq_cst.
Assisted-By: Claude
---
Patch is 29.88 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/241295.diff
12 Files Affected:
- (modified) clang/docs/ReleaseNotes.md (+4)
- (modified) clang/lib/CodeGen/CGOpenMPRuntime.cpp (+25-5)
- (modified) clang/lib/CodeGen/CGOpenMPRuntime.h (+2)
- (modified) clang/lib/CodeGen/CGStmtOpenMP.cpp (+21-16)
- (modified) clang/test/OpenMP/flush_codegen.cpp (+6-6)
- (added) clang/test/OpenMP/flush_memory_order_codegen.cpp (+92)
- (modified) llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h (+16-3)
- (modified) llvm/include/llvm/Frontend/OpenMP/OMPKinds.def (+3)
- (modified) llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp (+49-48)
- (modified) llvm/test/Transforms/OpenMP/add_attributes.ll (+8)
- (modified) llvm/unittests/Frontend/OpenMPIRBuilderTest.cpp (+89)
- (added) openmp/runtime/test/flush/omp_flush_memory_order.c (+41)
``````````diff
diff --git a/clang/docs/ReleaseNotes.md b/clang/docs/ReleaseNotes.md
index 0f9e7e01636d71..c0e602b72bd0d4 100644
--- a/clang/docs/ReleaseNotes.md
+++ b/clang/docs/ReleaseNotes.md
@@ -1588,6 +1588,10 @@ The `alpha.cplusplus.UseAfterLifetimeEnd` checker was renamed to `alpha.core.Use
- A loop transformation (`tile`, `unroll`, `interchange`, ...) that consumes
another tile's intra-tile loop.
+- The `acquire`, `release` and `acq_rel` clauses of the `flush` directive now
+ take effect. On GPUs a flush is emitted as a fence of the requested ordering.
+ On the host, a flush other than a seq_cst flush needs an OpenMP runtime from
+ this release.
- Added parsing and semantic support for `dims` modifier in `num_teams`,
`thread_limit` and `num_threads` clauses for OpenMP 6.1 or later.
- Map-type-modifying modifiers applied to a list item with a user-defined mapper
diff --git a/clang/lib/CodeGen/CGOpenMPRuntime.cpp b/clang/lib/CodeGen/CGOpenMPRuntime.cpp
index 62a537c885fc31..9e6d3e0c5a5199 100644
--- a/clang/lib/CodeGen/CGOpenMPRuntime.cpp
+++ b/clang/lib/CodeGen/CGOpenMPRuntime.cpp
@@ -2953,16 +2953,36 @@ void CGOpenMPRuntime::emitProcBindClause(CodeGenFunction &CGF,
void CGOpenMPRuntime::emitFlush(CodeGenFunction &CGF, ArrayRef<const Expr *>,
SourceLocation Loc, llvm::AtomicOrdering AO) {
- if (CGF.CGM.getLangOpts().OpenMPIRBuilder) {
- OMPBuilder.createFlush(CGF.Builder);
- } else {
- if (!CGF.HaveInsertPoint())
- return;
+ // The OpenMPIRBuilder emits a fence on a GPU, so GPUs always use it.
+ if (CGF.CGM.getLangOpts().OpenMPIRBuilder || isGPU()) {
+ OMPBuilder.createFlush(CGF.Builder, AO);
+ return;
+ }
+ if (!CGF.HaveInsertPoint())
+ return;
+ // This is the host lowering of OpenMPIRBuilder::emitFlush, but with the
+ // source location from emitUpdateLocation, as for clang's other host
+ // runtime calls.
+ // A flush with a list is a strong flush, which is treated as seq_cst.
+ if (AO == llvm::AtomicOrdering::NotAtomic)
+ AO = llvm::AtomicOrdering::SequentiallyConsistent;
+ if (AO == llvm::AtomicOrdering::SequentiallyConsistent) {
// Build call void __kmpc_flush(ident_t *loc)
CGF.EmitRuntimeCall(OMPBuilder.getOrCreateRuntimeFunction(
CGM.getModule(), OMPRTL___kmpc_flush),
emitUpdateLocation(CGF, Loc));
+ return;
}
+ // Build call void __kmpc_flush_explicit(ident_t *loc, kmp_int32 order,
+ // kmp_int32 scope), where order is an __ATOMIC_* value and scope is the
+ // __MEMORY_SCOPE_* value for device scope.
+ llvm::Value *Args[] = {
+ emitUpdateLocation(CGF, Loc),
+ CGF.Builder.getInt32(static_cast<unsigned>(llvm::toCABI(AO))),
+ CGF.Builder.getInt32(static_cast<unsigned>(llvm::AtomicScope::Device))};
+ CGF.EmitRuntimeCall(OMPBuilder.getOrCreateRuntimeFunction(
+ CGM.getModule(), OMPRTL___kmpc_flush_explicit),
+ Args);
}
namespace {
diff --git a/clang/lib/CodeGen/CGOpenMPRuntime.h b/clang/lib/CodeGen/CGOpenMPRuntime.h
index ae55295cadc14f..2f719aafccc9cc 100644
--- a/clang/lib/CodeGen/CGOpenMPRuntime.h
+++ b/clang/lib/CodeGen/CGOpenMPRuntime.h
@@ -1141,6 +1141,8 @@ class CGOpenMPRuntime {
/// Emit flush of the variables specified in 'omp flush' directive.
/// \param Vars List of variables to flush.
+ /// \param AO Memory ordering of the flush, or NotAtomic for a flush with a
+ /// list (a strong flush).
virtual void emitFlush(CodeGenFunction &CGF, ArrayRef<const Expr *> Vars,
SourceLocation Loc, llvm::AtomicOrdering AO);
diff --git a/clang/lib/CodeGen/CGStmtOpenMP.cpp b/clang/lib/CodeGen/CGStmtOpenMP.cpp
index 62fa820532fbd9..6959fe4ada619a 100644
--- a/clang/lib/CodeGen/CGStmtOpenMP.cpp
+++ b/clang/lib/CodeGen/CGStmtOpenMP.cpp
@@ -6262,9 +6262,17 @@ void CodeGenFunction::EmitOMPTaskgroupDirective(
}
void CodeGenFunction::EmitOMPFlushDirective(const OMPFlushDirective &S) {
- llvm::AtomicOrdering AO = S.getSingleClause<OMPFlushClause>()
- ? llvm::AtomicOrdering::NotAtomic
- : llvm::AtomicOrdering::AcquireRelease;
+ // A flush with a list is a strong flush. Without a list the flush behaves as
+ // if the memory-order clause is seq_cst unless another one is given.
+ llvm::AtomicOrdering AO = llvm::AtomicOrdering::SequentiallyConsistent;
+ if (S.getSingleClause<OMPFlushClause>())
+ AO = llvm::AtomicOrdering::NotAtomic;
+ else if (S.getSingleClause<OMPAcqRelClause>())
+ AO = llvm::AtomicOrdering::AcquireRelease;
+ else if (S.getSingleClause<OMPAcquireClause>())
+ AO = llvm::AtomicOrdering::Acquire;
+ else if (S.getSingleClause<OMPReleaseClause>())
+ AO = llvm::AtomicOrdering::Release;
CGM.getOpenMPRuntime().emitFlush(
*this,
[&S]() -> ArrayRef<const Expr *> {
@@ -6975,12 +6983,15 @@ static void emitOMPAtomicReadExpr(CodeGenFunction &CGF, llvm::AtomicOrdering AO,
// If the read or capture clause is specified and the acquire, acq_rel, or
// seq_cst clause is specified then the strong flush on exit from the atomic
// operation is also an acquire flush.
+ // The flushes implied by an atomic construct are all emitted after the
+ // atomic operation, where a release flush would order nothing, so they are
+ // emitted as seq_cst flushes.
switch (AO) {
case llvm::AtomicOrdering::Acquire:
case llvm::AtomicOrdering::AcquireRelease:
case llvm::AtomicOrdering::SequentiallyConsistent:
- CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
- llvm::AtomicOrdering::Acquire);
+ CGF.CGM.getOpenMPRuntime().emitFlush(
+ CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
break;
case llvm::AtomicOrdering::Monotonic:
case llvm::AtomicOrdering::Release:
@@ -7008,8 +7019,8 @@ static void emitOMPAtomicWriteExpr(CodeGenFunction &CGF,
case llvm::AtomicOrdering::Release:
case llvm::AtomicOrdering::AcquireRelease:
case llvm::AtomicOrdering::SequentiallyConsistent:
- CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
- llvm::AtomicOrdering::Release);
+ CGF.CGM.getOpenMPRuntime().emitFlush(
+ CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
break;
case llvm::AtomicOrdering::Acquire:
case llvm::AtomicOrdering::Monotonic:
@@ -7198,8 +7209,8 @@ static void emitOMPAtomicUpdateExpr(CodeGenFunction &CGF,
case llvm::AtomicOrdering::Release:
case llvm::AtomicOrdering::AcquireRelease:
case llvm::AtomicOrdering::SequentiallyConsistent:
- CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
- llvm::AtomicOrdering::Release);
+ CGF.CGM.getOpenMPRuntime().emitFlush(
+ CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
break;
case llvm::AtomicOrdering::Acquire:
case llvm::AtomicOrdering::Monotonic:
@@ -7313,17 +7324,11 @@ static void emitOMPAtomicCaptureExpr(CodeGenFunction &CGF,
// operation is also an acquire flush.
switch (AO) {
case llvm::AtomicOrdering::Release:
- CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
- llvm::AtomicOrdering::Release);
- break;
case llvm::AtomicOrdering::Acquire:
- CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
- llvm::AtomicOrdering::Acquire);
- break;
case llvm::AtomicOrdering::AcquireRelease:
case llvm::AtomicOrdering::SequentiallyConsistent:
CGF.CGM.getOpenMPRuntime().emitFlush(
- CGF, {}, Loc, llvm::AtomicOrdering::AcquireRelease);
+ CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
break;
case llvm::AtomicOrdering::Monotonic:
break;
diff --git a/clang/test/OpenMP/flush_codegen.cpp b/clang/test/OpenMP/flush_codegen.cpp
index 022bf856d82c91..195b0186b847f0 100644
--- a/clang/test/OpenMP/flush_codegen.cpp
+++ b/clang/test/OpenMP/flush_codegen.cpp
@@ -48,9 +48,9 @@ int main() {
#pragma omp flush(a)
// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
- // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
- // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
- // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
+ // CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 4, i32 1)
+ // CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 2, i32 1)
+ // CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 3, i32 1)
// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
return tmain(a);
// CHECK: call {{.*}} [[TMAIN:@.+]](
@@ -60,9 +60,9 @@ int main() {
// CHECK: [[TMAIN]]
// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
-// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
-// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
-// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
+// CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 4, i32 1)
+// CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 2, i32 1)
+// CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 3, i32 1)
// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
// CHECK: ret
diff --git a/clang/test/OpenMP/flush_memory_order_codegen.cpp b/clang/test/OpenMP/flush_memory_order_codegen.cpp
new file mode 100644
index 00000000000000..64357fe15a16ea
--- /dev/null
+++ b/clang/test/OpenMP/flush_memory_order_codegen.cpp
@@ -0,0 +1,92 @@
+// Check that the memory-order clauses of 'flush' reach the host runtime and
+// become fences on GPUs, and that the flushes implied by 'atomic' stay seq_cst.
+
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -o - | FileCheck %s --check-prefix=HOST
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -fopenmp-enable-irbuilder -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -o - | FileCheck %s --check-prefix=HOST
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm-bc %s -o %t-host.bc
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host.bc -o - | FileCheck %s --check-prefixes=DEV,AMDGPU
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -fopenmp-enable-irbuilder -x c++ -triple amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host.bc -o - | FileCheck %s --check-prefixes=DEV,AMDGPU
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=nvptx64-nvidia-cuda -emit-llvm-bc %s -o %t-host-nv.bc
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple nvptx64-nvidia-cuda -fopenmp-targets=nvptx64-nvidia-cuda -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host-nv.bc -o - | FileCheck %s --check-prefixes=DEV,NVPTX
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=spirv64-intel -emit-llvm-bc %s -o %t-host-spirv.bc
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple spirv64-intel -fopenmp-targets=spirv64-intel -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host-spirv.bc -o - | FileCheck %s --check-prefixes=DEV,SPIRV
+// expected-no-diagnostics
+
+int x, v, e, d;
+
+void foo() {
+#pragma omp target
+ {
+#pragma omp flush
+#pragma omp flush(x)
+#pragma omp flush seq_cst
+#pragma omp flush acq_rel
+#pragma omp flush acquire
+#pragma omp flush release
+#pragma omp atomic read acquire
+ v = x;
+#pragma omp atomic write release
+ x = v;
+#pragma omp atomic compare seq_cst
+ if (x == e)
+ x = d;
+ }
+}
+
+// The host keeps calling __kmpc_flush for a seq_cst flush, and passes any
+// other ordering to __kmpc_flush_explicit as an __ATOMIC_* value, with the
+// device scope as a __MEMORY_SCOPE_* value.
+// HOST-LABEL: define {{.*}}@{{.*}}foo{{.*}}(
+// HOST: call void @__kmpc_flush(ptr
+// HOST-NEXT: call void @__kmpc_flush(ptr
+// HOST-NEXT: call void @__kmpc_flush(ptr
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 4, i32 1)
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 2, i32 1)
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 3, i32 1)
+// HOST: load atomic i32, ptr {{.*}} acquire
+// HOST-NEXT: call void @__kmpc_flush(ptr
+// HOST: store atomic i32 {{.*}} release
+// HOST-NEXT: call void @__kmpc_flush(ptr
+// HOST: cmpxchg ptr {{.*}} seq_cst seq_cst
+// HOST: call void @__kmpc_flush(ptr
+
+// The device emits fences directly, at device scope.
+// DEV-LABEL: define {{.*}}@__omp_offloading_{{.*}}foo{{.*}}(
+// DEV-NOT: call {{.*}}@__kmpc_flush
+// AMDGPU: fence syncscope("agent") seq_cst
+// AMDGPU-NEXT: fence syncscope("agent") seq_cst
+// AMDGPU-NEXT: fence syncscope("agent") seq_cst
+// AMDGPU-NEXT: fence syncscope("agent") acq_rel
+// AMDGPU-NEXT: fence syncscope("agent") acquire
+// AMDGPU-NEXT: fence syncscope("agent") release
+// AMDGPU: load atomic i32, ptr {{.*}} acquire
+// AMDGPU-NEXT: fence syncscope("agent") seq_cst
+// AMDGPU: store atomic i32 {{.*}} release
+// AMDGPU-NEXT: fence syncscope("agent") seq_cst
+// AMDGPU: cmpxchg ptr {{.*}} seq_cst seq_cst
+// AMDGPU: fence syncscope("agent") seq_cst
+// NVPTX: fence syncscope("device") seq_cst
+// NVPTX-NEXT: fence syncscope("device") seq_cst
+// NVPTX-NEXT: fence syncscope("device") seq_cst
+// NVPTX-NEXT: fence syncscope("device") acq_rel
+// NVPTX-NEXT: fence syncscope("device") acquire
+// NVPTX-NEXT: fence syncscope("device") release
+// NVPTX: load atomic i32, ptr {{.*}} acquire
+// NVPTX-NEXT: fence syncscope("device") seq_cst
+// NVPTX: store atomic i32 {{.*}} release
+// NVPTX-NEXT: fence syncscope("device") seq_cst
+// NVPTX: cmpxchg ptr {{.*}} seq_cst seq_cst
+// NVPTX: fence syncscope("device") seq_cst
+// SPIRV: fence syncscope("device") seq_cst
+// SPIRV-NEXT: fence syncscope("device") seq_cst
+// SPIRV-NEXT: fence syncscope("device") seq_cst
+// SPIRV-NEXT: fence syncscope("device") acq_rel
+// SPIRV-NEXT: fence syncscope("device") acquire
+// SPIRV-NEXT: fence syncscope("device") release
+// SPIRV: load atomic i32, ptr {{.*}} acquire
+// SPIRV-NEXT: fence syncscope("device") seq_cst
+// SPIRV: store atomic i32 {{.*}} release
+// SPIRV-NEXT: fence syncscope("device") seq_cst
+// SPIRV: cmpxchg ptr {{.*}} seq_cst seq_cst
+// SPIRV: fence syncscope("device") seq_cst
+// DEV-NOT: call {{.*}}@__kmpc_flush
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
index ee83c02005cb9d..16fa4357b3b57f 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
@@ -28,6 +28,7 @@
#include "llvm/Support/Compiler.h"
#include "llvm/Support/Error.h"
#include "llvm/Support/VirtualFileSystemFwd.h"
+#include "llvm/TargetParser/AtomicScope.h"
#include "llvm/TargetParser/Triple.h"
#include <forward_list>
#include <map>
@@ -1595,7 +1596,13 @@ class OpenMPIRBuilder {
/// Generator for '#omp flush'
///
/// \param Loc The location where the flush directive was encountered
- LLVM_ABI void createFlush(const LocationDescription &Loc);
+ /// \param AO The memory ordering of the flush. NotAtomic is a flush with a
+ /// list, which is treated as seq_cst.
+ /// \param Scope The memory scope of the flush.
+ LLVM_ABI void
+ createFlush(const LocationDescription &Loc,
+ AtomicOrdering AO = AtomicOrdering::SequentiallyConsistent,
+ AtomicScope Scope = AtomicScope::Device);
/// Generate a call to the runtime to emit the diagnostic of an OpenMP
/// `error` directive with `at(execution)`.
@@ -2688,10 +2695,16 @@ class OpenMPIRBuilder {
Value *NumThreads, Value *HostPtr,
ArrayRef<Value *> KernelArgs);
- /// Generate a flush runtime call.
+ /// Generate a flush: a fence on a GPU, and a runtime call otherwise.
///
/// \param Loc The location at which the request originated and is fulfilled.
- LLVM_ABI void emitFlush(const LocationDescription &Loc);
+ /// \param AO The memory ordering of the flush. NotAtomic is a flush with a
+ /// list, which is treated as seq_cst.
+ /// \param Scope The memory scope of the flush.
+ LLVM_ABI void
+ emitFlush(const LocationDescription &Loc,
+ AtomicOrdering AO = AtomicOrdering::SequentiallyConsistent,
+ AtomicScope Scope = AtomicScope::Device);
/// The finalization stack made up of finalize callbacks currently in-flight,
/// wrapped into FinalizationInfo objects that reference also the finalization
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPKinds.def b/llvm/include/llvm/Frontend/OpenMP/OMPKinds.def
index f85379b1e80f82..3e6519f7b26a01 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPKinds.def
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPKinds.def
@@ -209,6 +209,7 @@ __OMP_RTL(__kmpc_cancel, false, Int32, IdentPtr, Int32, Int32)
__OMP_RTL(__kmpc_cancel_barrier, false, Int32, IdentPtr, Int32)
__OMP_RTL(__kmpc_error, false, Void, IdentPtr, Int32, Int8Ptr)
__OMP_RTL(__kmpc_flush, false, Void, IdentPtr)
+__OMP_RTL(__kmpc_flush_explicit, false, Void, IdentPtr, Int32, Int32)
__OMP_RTL(__kmpc_global_thread_num, false, Int32, IdentPtr)
__OMP_RTL(__kmpc_get_hardware_thread_id_in_block, false, Int32, )
__OMP_RTL(__kmpc_fork_call, true, Void, IdentPtr, Int32, ParallelTaskPtr)
@@ -703,6 +704,8 @@ __OMP_RTL_ATTRS(__kmpc_error, AttributeSet(), AttributeSet(),
ParamAttrs(AttributeSet(), SExt))
__OMP_RTL_ATTRS(__kmpc_flush, BarrierAttrs, AttributeSet(),
ParamAttrs(ReadOnlyPtrAttrs))
+__OMP_RTL_ATTRS(__kmpc_flush_explicit, BarrierAttrs, AttributeSet(),
+ ParamAttrs(ReadOnlyPtrAttrs, SExt, SExt))
__OMP_RTL_ATTRS(__kmpc_global_thread_num, GetterArgReadAttrs, SExt,
ParamAttrs(ReadOnlyPtrAttrs))
__OMP_RTL_ATTRS(__kmpc_get_hardware_thread_id_in_block, GetterAttrs, ZExt,
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index 81fc07f44177ca..d765b1a3d28d74 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -2329,20 +2329,45 @@ OpenMPIRBuilder::InsertPointOrErrorTy OpenMPIRBuilder::createParallel(
return AfterIP;
}
-void OpenMPIRBuilder::emitFlush(const LocationDescription &Loc) {
- // Build call void __kmpc_flush(ident_t *loc)
+void OpenMPIRBuilder::emitFlush(const LocationDescription &Loc,
+ AtomicOrdering AO, AtomicScope Scope) {
+ // A flush with a list is a strong flush, which is treated as seq_cst.
+ if (AO == AtomicOrdering::NotAtomic)
+ AO = AtomicOrdering::SequentiallyConsistent;
+
+ // A flush without a list corresponds to atomic_thread_fence, so on a GPU
+ // emit the fence directly rather than calling into the device runtime.
+ if (Config.IsGPU.value_or(M.getTargetTriple().isGPU())) {
+ StringRef ScopeName =
+ getAtomicScopeIRString(M.getTargetTriple(), Scope).value_or("");
+ Builder.CreateFence(AO, M.getContext().getOrInsertSyncScopeID(ScopeName));
+ return;
+ }
+
uint32_t SrcLocStrSize;
Constant *SrcLocStr = getOrCreateSrcLocStr(Loc, SrcLocStrSize);
- Value *Args[] = {getOrCreateIdent(SrcLocStr, SrcLocStrSize)};
-
- createRuntimeFunctionCall(getOrCreateRuntimeFunctionPtr(OMPRTL___kmpc_flush),
- Args);
+ Value *Ident = getOrCreateIdent(SrcLocStr, SrcLocStrSize);
+ if (AO == AtomicOrdering::SequentiallyConsistent &&
+ Scope == AtomicScope::Device) {
+ // Build call void __kmpc_flush(ident_t *loc)
+ createRuntimeFunctionCall(
+ getOrCreateRuntimeFunctionPtr(OMPRTL___kmpc_flush), {Ident});
+ return;
+ }
+ // Build call void __kmpc_flush_exp...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/241295
More information about the cfe-commits
mailing list