[clang] [llvm] [openmp] [OpenMP] Honour memory-order clauses on the flush directive (PR #241295)
Julian Brown via cfe-commits
cfe-commits at lists.llvm.org
Wed Nov 11 03:46:23 PST 2026
https://github.com/jtb20 created https://github.com/llvm/llvm-project/pull/241295
At present the acquire, release and acq_rel clauses on 'flush' are accepted but have no effect. Flushes are emitted as __kmpc_flush runtime calls, which implement a seq_cst fence.
AMD have had a downstream patch for a while (by Carlo Bertolli) which adds per-ordering __kmpc_flush_* entry points to the device runtime. This patch takes a slightly different approach: for the host, a seq_cst flush still calls __kmpc_flush, and a flush with any other ordering calls the __kmpc_flush_explicit entry point from #133325, so it needs a runtime that has that. For the GPU, it emits a fence directly.
This has mild benefits: OpenMPOpt already handles fences when it converts kernels to SPMD mode, but has no knowledge of the new __kmpc_flush calls (either _explicit or Carlo's per-ordering flush calls). Scope names differ per-GPU: they are determined using the mechanism introduced by
The lowering is done in OpenMPIRBuilder. The createFlush function now takes an ordering and a scope, and is now used for every GPU flush. The OpenMPIRBuilder-disabled path is unaffected. Flang/MLIR call createFlush with the defaults, so their host output is unchanged, but their GPU flushes, including those implied by atomics, now become fences instead of __kmpc_flush calls.
Since e8e05de08bea ("[OPENMP50]Add codegen for acq_rel clause in atomic|flush directives"), a flush with neither a list nor a memory-order clause has been passed to emitFlush as acq_rel. From OpenMP 5.1 it should behave as seq_cst instead, so it is now emitted as that.
Flushes implied by 'atomic' are also emitted as seq_cst.
Assisted-By: Claude
>From 9cf423acb97ed94a071cce9b524cc54c73f54444 Mon Sep 17 00:00:00 2001
From: Julian Brown <julian.brown at amd.com>
Date: Wed, 4 Nov 2026 06:59:20 -0600
Subject: [PATCH] [OpenMP] Honour memory-order clauses on the flush directive
At present the acquire, release and acq_rel clauses on 'flush' are
accepted but have no effect. Flushes are emitted as __kmpc_flush runtime
calls, which implement a seq_cst fence.
AMD have had a downstream patch for a while (by Carlo Bertolli) which
adds per-ordering __kmpc_flush_* entry points to the device runtime.
This patch takes a slightly different approach: for the host, a seq_cst
flush still calls __kmpc_flush, and a flush with any other ordering
calls the __kmpc_flush_explicit entry point from #133325, so it needs
a runtime that has that. For the GPU, it emits a fence directly.
This has mild benefits: OpenMPOpt already handles fences when it converts
kernels to SPMD mode, but has no knowledge of the new __kmpc_flush calls
(either _explicit or Carlo's per-ordering flush calls). Scope names
differ per-GPU: they are determined using the mechanism introduced by
The lowering is done in OpenMPIRBuilder. The createFlush function now
takes an ordering and a scope, and is now used for every GPU flush.
The OpenMPIRBuilder-disabled path is unaffected. Flang/MLIR call
createFlush with the defaults, so their host output is unchanged, but
their GPU flushes, including those implied by atomics, now become fences
instead of __kmpc_flush calls.
Since e8e05de08bea ("[OPENMP50]Add codegen for acq_rel clause in
atomic|flush directives"), a flush with neither a list nor a memory-order
clause has been passed to emitFlush as acq_rel. From OpenMP 5.1 it should
behave as seq_cst instead, so it is now emitted as that.
Flushes implied by 'atomic' are also emitted as seq_cst.
Co-authored-by: Carlo Bertolli <Carlo.Bertolli at amd.com>
Assisted-By: Claude
---
clang/docs/ReleaseNotes.md | 4 +
clang/lib/CodeGen/CGOpenMPRuntime.cpp | 30 +++++-
clang/lib/CodeGen/CGOpenMPRuntime.h | 2 +
clang/lib/CodeGen/CGStmtOpenMP.cpp | 37 ++++---
clang/test/OpenMP/flush_codegen.cpp | 12 +--
.../OpenMP/flush_memory_order_codegen.cpp | 92 ++++++++++++++++++
.../llvm/Frontend/OpenMP/OMPIRBuilder.h | 19 +++-
.../include/llvm/Frontend/OpenMP/OMPKinds.def | 3 +
llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp | 97 ++++++++++---------
llvm/test/Transforms/OpenMP/add_attributes.ll | 8 ++
.../Frontend/OpenMPIRBuilderTest.cpp | 89 +++++++++++++++++
.../test/flush/omp_flush_memory_order.c | 41 ++++++++
12 files changed, 356 insertions(+), 78 deletions(-)
create mode 100644 clang/test/OpenMP/flush_memory_order_codegen.cpp
create mode 100644 openmp/runtime/test/flush/omp_flush_memory_order.c
diff --git a/clang/docs/ReleaseNotes.md b/clang/docs/ReleaseNotes.md
index 0f9e7e01636d71..c0e602b72bd0d4 100644
--- a/clang/docs/ReleaseNotes.md
+++ b/clang/docs/ReleaseNotes.md
@@ -1588,6 +1588,10 @@ The `alpha.cplusplus.UseAfterLifetimeEnd` checker was renamed to `alpha.core.Use
- A loop transformation (`tile`, `unroll`, `interchange`, ...) that consumes
another tile's intra-tile loop.
+- The `acquire`, `release` and `acq_rel` clauses of the `flush` directive now
+ take effect. On GPUs a flush is emitted as a fence of the requested ordering.
+ On the host, a flush other than a seq_cst flush needs an OpenMP runtime from
+ this release.
- Added parsing and semantic support for `dims` modifier in `num_teams`,
`thread_limit` and `num_threads` clauses for OpenMP 6.1 or later.
- Map-type-modifying modifiers applied to a list item with a user-defined mapper
diff --git a/clang/lib/CodeGen/CGOpenMPRuntime.cpp b/clang/lib/CodeGen/CGOpenMPRuntime.cpp
index 62a537c885fc31..9e6d3e0c5a5199 100644
--- a/clang/lib/CodeGen/CGOpenMPRuntime.cpp
+++ b/clang/lib/CodeGen/CGOpenMPRuntime.cpp
@@ -2953,16 +2953,36 @@ void CGOpenMPRuntime::emitProcBindClause(CodeGenFunction &CGF,
void CGOpenMPRuntime::emitFlush(CodeGenFunction &CGF, ArrayRef<const Expr *>,
SourceLocation Loc, llvm::AtomicOrdering AO) {
- if (CGF.CGM.getLangOpts().OpenMPIRBuilder) {
- OMPBuilder.createFlush(CGF.Builder);
- } else {
- if (!CGF.HaveInsertPoint())
- return;
+ // The OpenMPIRBuilder emits a fence on a GPU, so GPUs always use it.
+ if (CGF.CGM.getLangOpts().OpenMPIRBuilder || isGPU()) {
+ OMPBuilder.createFlush(CGF.Builder, AO);
+ return;
+ }
+ if (!CGF.HaveInsertPoint())
+ return;
+ // This is the host lowering of OpenMPIRBuilder::emitFlush, but with the
+ // source location from emitUpdateLocation, as for clang's other host
+ // runtime calls.
+ // A flush with a list is a strong flush, which is treated as seq_cst.
+ if (AO == llvm::AtomicOrdering::NotAtomic)
+ AO = llvm::AtomicOrdering::SequentiallyConsistent;
+ if (AO == llvm::AtomicOrdering::SequentiallyConsistent) {
// Build call void __kmpc_flush(ident_t *loc)
CGF.EmitRuntimeCall(OMPBuilder.getOrCreateRuntimeFunction(
CGM.getModule(), OMPRTL___kmpc_flush),
emitUpdateLocation(CGF, Loc));
+ return;
}
+ // Build call void __kmpc_flush_explicit(ident_t *loc, kmp_int32 order,
+ // kmp_int32 scope), where order is an __ATOMIC_* value and scope is the
+ // __MEMORY_SCOPE_* value for device scope.
+ llvm::Value *Args[] = {
+ emitUpdateLocation(CGF, Loc),
+ CGF.Builder.getInt32(static_cast<unsigned>(llvm::toCABI(AO))),
+ CGF.Builder.getInt32(static_cast<unsigned>(llvm::AtomicScope::Device))};
+ CGF.EmitRuntimeCall(OMPBuilder.getOrCreateRuntimeFunction(
+ CGM.getModule(), OMPRTL___kmpc_flush_explicit),
+ Args);
}
namespace {
diff --git a/clang/lib/CodeGen/CGOpenMPRuntime.h b/clang/lib/CodeGen/CGOpenMPRuntime.h
index ae55295cadc14f..2f719aafccc9cc 100644
--- a/clang/lib/CodeGen/CGOpenMPRuntime.h
+++ b/clang/lib/CodeGen/CGOpenMPRuntime.h
@@ -1141,6 +1141,8 @@ class CGOpenMPRuntime {
/// Emit flush of the variables specified in 'omp flush' directive.
/// \param Vars List of variables to flush.
+ /// \param AO Memory ordering of the flush, or NotAtomic for a flush with a
+ /// list (a strong flush).
virtual void emitFlush(CodeGenFunction &CGF, ArrayRef<const Expr *> Vars,
SourceLocation Loc, llvm::AtomicOrdering AO);
diff --git a/clang/lib/CodeGen/CGStmtOpenMP.cpp b/clang/lib/CodeGen/CGStmtOpenMP.cpp
index 62fa820532fbd9..6959fe4ada619a 100644
--- a/clang/lib/CodeGen/CGStmtOpenMP.cpp
+++ b/clang/lib/CodeGen/CGStmtOpenMP.cpp
@@ -6262,9 +6262,17 @@ void CodeGenFunction::EmitOMPTaskgroupDirective(
}
void CodeGenFunction::EmitOMPFlushDirective(const OMPFlushDirective &S) {
- llvm::AtomicOrdering AO = S.getSingleClause<OMPFlushClause>()
- ? llvm::AtomicOrdering::NotAtomic
- : llvm::AtomicOrdering::AcquireRelease;
+ // A flush with a list is a strong flush. Without a list the flush behaves as
+ // if the memory-order clause is seq_cst unless another one is given.
+ llvm::AtomicOrdering AO = llvm::AtomicOrdering::SequentiallyConsistent;
+ if (S.getSingleClause<OMPFlushClause>())
+ AO = llvm::AtomicOrdering::NotAtomic;
+ else if (S.getSingleClause<OMPAcqRelClause>())
+ AO = llvm::AtomicOrdering::AcquireRelease;
+ else if (S.getSingleClause<OMPAcquireClause>())
+ AO = llvm::AtomicOrdering::Acquire;
+ else if (S.getSingleClause<OMPReleaseClause>())
+ AO = llvm::AtomicOrdering::Release;
CGM.getOpenMPRuntime().emitFlush(
*this,
[&S]() -> ArrayRef<const Expr *> {
@@ -6975,12 +6983,15 @@ static void emitOMPAtomicReadExpr(CodeGenFunction &CGF, llvm::AtomicOrdering AO,
// If the read or capture clause is specified and the acquire, acq_rel, or
// seq_cst clause is specified then the strong flush on exit from the atomic
// operation is also an acquire flush.
+ // The flushes implied by an atomic construct are all emitted after the
+ // atomic operation, where a release flush would order nothing, so they are
+ // emitted as seq_cst flushes.
switch (AO) {
case llvm::AtomicOrdering::Acquire:
case llvm::AtomicOrdering::AcquireRelease:
case llvm::AtomicOrdering::SequentiallyConsistent:
- CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
- llvm::AtomicOrdering::Acquire);
+ CGF.CGM.getOpenMPRuntime().emitFlush(
+ CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
break;
case llvm::AtomicOrdering::Monotonic:
case llvm::AtomicOrdering::Release:
@@ -7008,8 +7019,8 @@ static void emitOMPAtomicWriteExpr(CodeGenFunction &CGF,
case llvm::AtomicOrdering::Release:
case llvm::AtomicOrdering::AcquireRelease:
case llvm::AtomicOrdering::SequentiallyConsistent:
- CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
- llvm::AtomicOrdering::Release);
+ CGF.CGM.getOpenMPRuntime().emitFlush(
+ CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
break;
case llvm::AtomicOrdering::Acquire:
case llvm::AtomicOrdering::Monotonic:
@@ -7198,8 +7209,8 @@ static void emitOMPAtomicUpdateExpr(CodeGenFunction &CGF,
case llvm::AtomicOrdering::Release:
case llvm::AtomicOrdering::AcquireRelease:
case llvm::AtomicOrdering::SequentiallyConsistent:
- CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
- llvm::AtomicOrdering::Release);
+ CGF.CGM.getOpenMPRuntime().emitFlush(
+ CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
break;
case llvm::AtomicOrdering::Acquire:
case llvm::AtomicOrdering::Monotonic:
@@ -7313,17 +7324,11 @@ static void emitOMPAtomicCaptureExpr(CodeGenFunction &CGF,
// operation is also an acquire flush.
switch (AO) {
case llvm::AtomicOrdering::Release:
- CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
- llvm::AtomicOrdering::Release);
- break;
case llvm::AtomicOrdering::Acquire:
- CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
- llvm::AtomicOrdering::Acquire);
- break;
case llvm::AtomicOrdering::AcquireRelease:
case llvm::AtomicOrdering::SequentiallyConsistent:
CGF.CGM.getOpenMPRuntime().emitFlush(
- CGF, {}, Loc, llvm::AtomicOrdering::AcquireRelease);
+ CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
break;
case llvm::AtomicOrdering::Monotonic:
break;
diff --git a/clang/test/OpenMP/flush_codegen.cpp b/clang/test/OpenMP/flush_codegen.cpp
index 022bf856d82c91..195b0186b847f0 100644
--- a/clang/test/OpenMP/flush_codegen.cpp
+++ b/clang/test/OpenMP/flush_codegen.cpp
@@ -48,9 +48,9 @@ int main() {
#pragma omp flush(a)
// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
- // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
- // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
- // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
+ // CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 4, i32 1)
+ // CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 2, i32 1)
+ // CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 3, i32 1)
// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
return tmain(a);
// CHECK: call {{.*}} [[TMAIN:@.+]](
@@ -60,9 +60,9 @@ int main() {
// CHECK: [[TMAIN]]
// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
-// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
-// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
-// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
+// CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 4, i32 1)
+// CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 2, i32 1)
+// CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 3, i32 1)
// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
// CHECK: ret
diff --git a/clang/test/OpenMP/flush_memory_order_codegen.cpp b/clang/test/OpenMP/flush_memory_order_codegen.cpp
new file mode 100644
index 00000000000000..64357fe15a16ea
--- /dev/null
+++ b/clang/test/OpenMP/flush_memory_order_codegen.cpp
@@ -0,0 +1,92 @@
+// Check that the memory-order clauses of 'flush' reach the host runtime and
+// become fences on GPUs, and that the flushes implied by 'atomic' stay seq_cst.
+
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -o - | FileCheck %s --check-prefix=HOST
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -fopenmp-enable-irbuilder -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -o - | FileCheck %s --check-prefix=HOST
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm-bc %s -o %t-host.bc
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host.bc -o - | FileCheck %s --check-prefixes=DEV,AMDGPU
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -fopenmp-enable-irbuilder -x c++ -triple amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host.bc -o - | FileCheck %s --check-prefixes=DEV,AMDGPU
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=nvptx64-nvidia-cuda -emit-llvm-bc %s -o %t-host-nv.bc
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple nvptx64-nvidia-cuda -fopenmp-targets=nvptx64-nvidia-cuda -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host-nv.bc -o - | FileCheck %s --check-prefixes=DEV,NVPTX
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=spirv64-intel -emit-llvm-bc %s -o %t-host-spirv.bc
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple spirv64-intel -fopenmp-targets=spirv64-intel -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host-spirv.bc -o - | FileCheck %s --check-prefixes=DEV,SPIRV
+// expected-no-diagnostics
+
+int x, v, e, d;
+
+void foo() {
+#pragma omp target
+ {
+#pragma omp flush
+#pragma omp flush(x)
+#pragma omp flush seq_cst
+#pragma omp flush acq_rel
+#pragma omp flush acquire
+#pragma omp flush release
+#pragma omp atomic read acquire
+ v = x;
+#pragma omp atomic write release
+ x = v;
+#pragma omp atomic compare seq_cst
+ if (x == e)
+ x = d;
+ }
+}
+
+// The host keeps calling __kmpc_flush for a seq_cst flush, and passes any
+// other ordering to __kmpc_flush_explicit as an __ATOMIC_* value, with the
+// device scope as a __MEMORY_SCOPE_* value.
+// HOST-LABEL: define {{.*}}@{{.*}}foo{{.*}}(
+// HOST: call void @__kmpc_flush(ptr
+// HOST-NEXT: call void @__kmpc_flush(ptr
+// HOST-NEXT: call void @__kmpc_flush(ptr
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 4, i32 1)
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 2, i32 1)
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 3, i32 1)
+// HOST: load atomic i32, ptr {{.*}} acquire
+// HOST-NEXT: call void @__kmpc_flush(ptr
+// HOST: store atomic i32 {{.*}} release
+// HOST-NEXT: call void @__kmpc_flush(ptr
+// HOST: cmpxchg ptr {{.*}} seq_cst seq_cst
+// HOST: call void @__kmpc_flush(ptr
+
+// The device emits fences directly, at device scope.
+// DEV-LABEL: define {{.*}}@__omp_offloading_{{.*}}foo{{.*}}(
+// DEV-NOT: call {{.*}}@__kmpc_flush
+// AMDGPU: fence syncscope("agent") seq_cst
+// AMDGPU-NEXT: fence syncscope("agent") seq_cst
+// AMDGPU-NEXT: fence syncscope("agent") seq_cst
+// AMDGPU-NEXT: fence syncscope("agent") acq_rel
+// AMDGPU-NEXT: fence syncscope("agent") acquire
+// AMDGPU-NEXT: fence syncscope("agent") release
+// AMDGPU: load atomic i32, ptr {{.*}} acquire
+// AMDGPU-NEXT: fence syncscope("agent") seq_cst
+// AMDGPU: store atomic i32 {{.*}} release
+// AMDGPU-NEXT: fence syncscope("agent") seq_cst
+// AMDGPU: cmpxchg ptr {{.*}} seq_cst seq_cst
+// AMDGPU: fence syncscope("agent") seq_cst
+// NVPTX: fence syncscope("device") seq_cst
+// NVPTX-NEXT: fence syncscope("device") seq_cst
+// NVPTX-NEXT: fence syncscope("device") seq_cst
+// NVPTX-NEXT: fence syncscope("device") acq_rel
+// NVPTX-NEXT: fence syncscope("device") acquire
+// NVPTX-NEXT: fence syncscope("device") release
+// NVPTX: load atomic i32, ptr {{.*}} acquire
+// NVPTX-NEXT: fence syncscope("device") seq_cst
+// NVPTX: store atomic i32 {{.*}} release
+// NVPTX-NEXT: fence syncscope("device") seq_cst
+// NVPTX: cmpxchg ptr {{.*}} seq_cst seq_cst
+// NVPTX: fence syncscope("device") seq_cst
+// SPIRV: fence syncscope("device") seq_cst
+// SPIRV-NEXT: fence syncscope("device") seq_cst
+// SPIRV-NEXT: fence syncscope("device") seq_cst
+// SPIRV-NEXT: fence syncscope("device") acq_rel
+// SPIRV-NEXT: fence syncscope("device") acquire
+// SPIRV-NEXT: fence syncscope("device") release
+// SPIRV: load atomic i32, ptr {{.*}} acquire
+// SPIRV-NEXT: fence syncscope("device") seq_cst
+// SPIRV: store atomic i32 {{.*}} release
+// SPIRV-NEXT: fence syncscope("device") seq_cst
+// SPIRV: cmpxchg ptr {{.*}} seq_cst seq_cst
+// SPIRV: fence syncscope("device") seq_cst
+// DEV-NOT: call {{.*}}@__kmpc_flush
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
index ee83c02005cb9d..16fa4357b3b57f 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
@@ -28,6 +28,7 @@
#include "llvm/Support/Compiler.h"
#include "llvm/Support/Error.h"
#include "llvm/Support/VirtualFileSystemFwd.h"
+#include "llvm/TargetParser/AtomicScope.h"
#include "llvm/TargetParser/Triple.h"
#include <forward_list>
#include <map>
@@ -1595,7 +1596,13 @@ class OpenMPIRBuilder {
/// Generator for '#omp flush'
///
/// \param Loc The location where the flush directive was encountered
- LLVM_ABI void createFlush(const LocationDescription &Loc);
+ /// \param AO The memory ordering of the flush. NotAtomic is a flush with a
+ /// list, which is treated as seq_cst.
+ /// \param Scope The memory scope of the flush.
+ LLVM_ABI void
+ createFlush(const LocationDescription &Loc,
+ AtomicOrdering AO = AtomicOrdering::SequentiallyConsistent,
+ AtomicScope Scope = AtomicScope::Device);
/// Generate a call to the runtime to emit the diagnostic of an OpenMP
/// `error` directive with `at(execution)`.
@@ -2688,10 +2695,16 @@ class OpenMPIRBuilder {
Value *NumThreads, Value *HostPtr,
ArrayRef<Value *> KernelArgs);
- /// Generate a flush runtime call.
+ /// Generate a flush: a fence on a GPU, and a runtime call otherwise.
///
/// \param Loc The location at which the request originated and is fulfilled.
- LLVM_ABI void emitFlush(const LocationDescription &Loc);
+ /// \param AO The memory ordering of the flush. NotAtomic is a flush with a
+ /// list, which is treated as seq_cst.
+ /// \param Scope The memory scope of the flush.
+ LLVM_ABI void
+ emitFlush(const LocationDescription &Loc,
+ AtomicOrdering AO = AtomicOrdering::SequentiallyConsistent,
+ AtomicScope Scope = AtomicScope::Device);
/// The finalization stack made up of finalize callbacks currently in-flight,
/// wrapped into FinalizationInfo objects that reference also the finalization
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPKinds.def b/llvm/include/llvm/Frontend/OpenMP/OMPKinds.def
index f85379b1e80f82..3e6519f7b26a01 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPKinds.def
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPKinds.def
@@ -209,6 +209,7 @@ __OMP_RTL(__kmpc_cancel, false, Int32, IdentPtr, Int32, Int32)
__OMP_RTL(__kmpc_cancel_barrier, false, Int32, IdentPtr, Int32)
__OMP_RTL(__kmpc_error, false, Void, IdentPtr, Int32, Int8Ptr)
__OMP_RTL(__kmpc_flush, false, Void, IdentPtr)
+__OMP_RTL(__kmpc_flush_explicit, false, Void, IdentPtr, Int32, Int32)
__OMP_RTL(__kmpc_global_thread_num, false, Int32, IdentPtr)
__OMP_RTL(__kmpc_get_hardware_thread_id_in_block, false, Int32, )
__OMP_RTL(__kmpc_fork_call, true, Void, IdentPtr, Int32, ParallelTaskPtr)
@@ -703,6 +704,8 @@ __OMP_RTL_ATTRS(__kmpc_error, AttributeSet(), AttributeSet(),
ParamAttrs(AttributeSet(), SExt))
__OMP_RTL_ATTRS(__kmpc_flush, BarrierAttrs, AttributeSet(),
ParamAttrs(ReadOnlyPtrAttrs))
+__OMP_RTL_ATTRS(__kmpc_flush_explicit, BarrierAttrs, AttributeSet(),
+ ParamAttrs(ReadOnlyPtrAttrs, SExt, SExt))
__OMP_RTL_ATTRS(__kmpc_global_thread_num, GetterArgReadAttrs, SExt,
ParamAttrs(ReadOnlyPtrAttrs))
__OMP_RTL_ATTRS(__kmpc_get_hardware_thread_id_in_block, GetterAttrs, ZExt,
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index 81fc07f44177ca..d765b1a3d28d74 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -2329,20 +2329,45 @@ OpenMPIRBuilder::InsertPointOrErrorTy OpenMPIRBuilder::createParallel(
return AfterIP;
}
-void OpenMPIRBuilder::emitFlush(const LocationDescription &Loc) {
- // Build call void __kmpc_flush(ident_t *loc)
+void OpenMPIRBuilder::emitFlush(const LocationDescription &Loc,
+ AtomicOrdering AO, AtomicScope Scope) {
+ // A flush with a list is a strong flush, which is treated as seq_cst.
+ if (AO == AtomicOrdering::NotAtomic)
+ AO = AtomicOrdering::SequentiallyConsistent;
+
+ // A flush without a list corresponds to atomic_thread_fence, so on a GPU
+ // emit the fence directly rather than calling into the device runtime.
+ if (Config.IsGPU.value_or(M.getTargetTriple().isGPU())) {
+ StringRef ScopeName =
+ getAtomicScopeIRString(M.getTargetTriple(), Scope).value_or("");
+ Builder.CreateFence(AO, M.getContext().getOrInsertSyncScopeID(ScopeName));
+ return;
+ }
+
uint32_t SrcLocStrSize;
Constant *SrcLocStr = getOrCreateSrcLocStr(Loc, SrcLocStrSize);
- Value *Args[] = {getOrCreateIdent(SrcLocStr, SrcLocStrSize)};
-
- createRuntimeFunctionCall(getOrCreateRuntimeFunctionPtr(OMPRTL___kmpc_flush),
- Args);
+ Value *Ident = getOrCreateIdent(SrcLocStr, SrcLocStrSize);
+ if (AO == AtomicOrdering::SequentiallyConsistent &&
+ Scope == AtomicScope::Device) {
+ // Build call void __kmpc_flush(ident_t *loc)
+ createRuntimeFunctionCall(
+ getOrCreateRuntimeFunctionPtr(OMPRTL___kmpc_flush), {Ident});
+ return;
+ }
+ // Build call void __kmpc_flush_explicit(ident_t *loc, kmp_int32 order,
+ // kmp_int32 scope), where order is an __ATOMIC_* value and scope is a
+ // __MEMORY_SCOPE_* value.
+ Value *Args[] = {Ident, Builder.getInt32(static_cast<unsigned>(toCABI(AO))),
+ Builder.getInt32(static_cast<unsigned>(Scope))};
+ createRuntimeFunctionCall(
+ getOrCreateRuntimeFunctionPtr(OMPRTL___kmpc_flush_explicit), Args);
}
-void OpenMPIRBuilder::createFlush(const LocationDescription &Loc) {
+void OpenMPIRBuilder::createFlush(const LocationDescription &Loc,
+ AtomicOrdering AO, AtomicScope Scope) {
if (!updateToLocation(Loc))
return;
- emitFlush(Loc);
+ emitFlush(Loc, AO, Scope);
}
void OpenMPIRBuilder::createError(const LocationDescription &Loc, bool IsFatal,
@@ -11548,57 +11573,33 @@ bool OpenMPIRBuilder::checkAndEmitFlushAfterAtomic(
"Unexpected Atomic Ordering.");
bool Flush = false;
- llvm::AtomicOrdering FlushAO = AtomicOrdering::Monotonic;
-
switch (AK) {
case Read:
- if (AO == AtomicOrdering::Acquire || AO == AtomicOrdering::AcquireRelease ||
- AO == AtomicOrdering::SequentiallyConsistent) {
- FlushAO = AtomicOrdering::Acquire;
- Flush = true;
- }
+ // The acquire flush on exit.
+ Flush = AO == AtomicOrdering::Acquire ||
+ AO == AtomicOrdering::AcquireRelease ||
+ AO == AtomicOrdering::SequentiallyConsistent;
break;
case Write:
case Compare:
case Update:
- if (AO == AtomicOrdering::Release || AO == AtomicOrdering::AcquireRelease ||
- AO == AtomicOrdering::SequentiallyConsistent) {
- FlushAO = AtomicOrdering::Release;
- Flush = true;
- }
+ // The release flush on entry.
+ Flush = AO == AtomicOrdering::Release ||
+ AO == AtomicOrdering::AcquireRelease ||
+ AO == AtomicOrdering::SequentiallyConsistent;
break;
case Capture:
- switch (AO) {
- case AtomicOrdering::Acquire:
- FlushAO = AtomicOrdering::Acquire;
- Flush = true;
- break;
- case AtomicOrdering::Release:
- FlushAO = AtomicOrdering::Release;
- Flush = true;
- break;
- case AtomicOrdering::AcquireRelease:
- case AtomicOrdering::SequentiallyConsistent:
- FlushAO = AtomicOrdering::AcquireRelease;
- Flush = true;
- break;
- default:
- // do nothing - leave silently.
- break;
- }
+ Flush = AO == AtomicOrdering::Acquire || AO == AtomicOrdering::Release ||
+ AO == AtomicOrdering::AcquireRelease ||
+ AO == AtomicOrdering::SequentiallyConsistent;
+ break;
}
- if (Flush) {
- // Currently Flush RT call still doesn't take memory_ordering, so for when
- // that happens, this tries to do the resolution of which atomic ordering
- // to use with but issue the flush call
- // TODO: pass `FlushAO` after memory ordering support is added
- (void)FlushAO;
+ // The flush is emitted after the atomic operation, even the release flush
+ // that belongs on entry, and a release fence there would order nothing. So
+ // the flush is seq_cst whichever flush the construct implies.
+ if (Flush)
emitFlush(Loc);
- }
-
- // for AO == AtomicOrdering::Monotonic and all other case combinations
- // do nothing
return Flush;
}
diff --git a/llvm/test/Transforms/OpenMP/add_attributes.ll b/llvm/test/Transforms/OpenMP/add_attributes.ll
index 3665da1fc17027..7011ab80a766c1 100644
--- a/llvm/test/Transforms/OpenMP/add_attributes.ll
+++ b/llvm/test/Transforms/OpenMP/add_attributes.ll
@@ -469,6 +469,8 @@ declare i32 @__kmpc_cancel_barrier(ptr, i32)
declare void @__kmpc_flush(ptr)
+declare void @__kmpc_flush_explicit(ptr, i32, i32)
+
declare i32 @__kmpc_global_thread_num(ptr)
declare void @__kmpc_fork_call(ptr, i32, ptr, ...)
@@ -994,6 +996,9 @@ declare i32 @__tgt_target_kernel_nowait(ptr, i64, i32, i32, ptr, ptr, i32, ptr,
; CHECK: ; Function Attrs: convergent nounwind
; CHECK-NEXT: declare void @__kmpc_flush(ptr)
+; CHECK: ; Function Attrs: convergent nounwind
+; CHECK-NEXT: declare void @__kmpc_flush_explicit(ptr, i32, i32)
+
; CHECK: ; Function Attrs: nounwind
; CHECK-NEXT: declare i32 @__kmpc_global_thread_num(ptr)
@@ -2303,6 +2308,9 @@ declare i32 @__tgt_target_kernel_nowait(ptr, i64, i32, i32, ptr, ptr, i32, ptr,
; EXT: ; Function Attrs: convergent nounwind
; EXT-NEXT: declare void @__kmpc_flush(ptr)
+; EXT: ; Function Attrs: convergent nounwind
+; EXT-NEXT: declare void @__kmpc_flush_explicit(ptr, i32 signext, i32 signext)
+
; EXT: ; Function Attrs: nounwind
; EXT-NEXT: declare signext i32 @__kmpc_global_thread_num(ptr)
diff --git a/llvm/unittests/Frontend/OpenMPIRBuilderTest.cpp b/llvm/unittests/Frontend/OpenMPIRBuilderTest.cpp
index 45a244a1d1ab31..1ec0f4c232a8bc 100644
--- a/llvm/unittests/Frontend/OpenMPIRBuilderTest.cpp
+++ b/llvm/unittests/Frontend/OpenMPIRBuilderTest.cpp
@@ -370,6 +370,95 @@ static Value *findAggregateFromValue(Value *V) {
return Aggregate;
}
+TEST_F(OpenMPIRBuilderTest, CreateFlush) {
+ OpenMPIRBuilder OMPBuilder(*M);
+ OMPBuilder.initialize();
+
+ IRBuilder<> Builder(BB);
+ OpenMPIRBuilder::LocationDescription Loc(Builder);
+ // A seq_cst flush at device scope, or a flush with a list, is a call to
+ // __kmpc_flush. Anything else calls __kmpc_flush_explicit with the
+ // __ATOMIC_* ordering and the __MEMORY_SCOPE_* scope.
+ OMPBuilder.createFlush(Loc);
+ OMPBuilder.createFlush(Loc, AtomicOrdering::NotAtomic);
+ OMPBuilder.createFlush(Loc, AtomicOrdering::Acquire);
+ OMPBuilder.createFlush(Loc, AtomicOrdering::SequentiallyConsistent,
+ AtomicScope::System);
+ Builder.CreateRetVoid();
+ EXPECT_FALSE(verifyModule(*M, &errs()));
+
+ SmallVector<CallInst *> Calls;
+ for (Instruction &I : *BB)
+ if (auto *CI = dyn_cast<CallInst>(&I))
+ Calls.push_back(CI);
+ ASSERT_EQ(Calls.size(), 4U);
+ EXPECT_EQ(Calls[0]->getCalledFunction()->getName(), "__kmpc_flush");
+ EXPECT_EQ(Calls[1]->getCalledFunction()->getName(), "__kmpc_flush");
+ for (auto [CI, Order, Scope] :
+ {std::tuple(Calls[2], 2, 1), std::tuple(Calls[3], 5, 0)}) {
+ EXPECT_EQ(CI->getCalledFunction()->getName(), "__kmpc_flush_explicit");
+ EXPECT_EQ(cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue(),
+ uint64_t(Order));
+ EXPECT_EQ(cast<ConstantInt>(CI->getArgOperand(2))->getZExtValue(),
+ uint64_t(Scope));
+ }
+}
+
+TEST_F(OpenMPIRBuilderTest, CreateFlushGPU) {
+ M->setTargetTriple(Triple("amdgcn-amd-amdhsa"));
+ OpenMPIRBuilder OMPBuilder(*M);
+ OMPBuilder.Config.IsTargetDevice = true;
+ OMPBuilder.Config.setIsGPU(true);
+ OMPBuilder.initialize();
+
+ IRBuilder<> Builder(BB);
+ OpenMPIRBuilder::LocationDescription Loc(Builder);
+ // On a GPU a flush is a fence of the given ordering and scope.
+ OMPBuilder.createFlush(Loc);
+ OMPBuilder.createFlush(Loc, AtomicOrdering::NotAtomic);
+ OMPBuilder.createFlush(Loc, AtomicOrdering::Release, AtomicScope::System);
+ OMPBuilder.createFlush(Loc, AtomicOrdering::Acquire, AtomicScope::Workgroup);
+ Builder.CreateRetVoid();
+ EXPECT_FALSE(verifyModule(*M, &errs()));
+
+ SmallVector<FenceInst *> Fences;
+ for (Instruction &I : *BB) {
+ EXPECT_FALSE(isa<CallInst>(I));
+ if (auto *FI = dyn_cast<FenceInst>(&I))
+ Fences.push_back(FI);
+ }
+ ASSERT_EQ(Fences.size(), 4U);
+ SyncScope::ID Agent = M->getContext().getOrInsertSyncScopeID("agent");
+ EXPECT_EQ(Fences[0]->getOrdering(), AtomicOrdering::SequentiallyConsistent);
+ EXPECT_EQ(Fences[0]->getSyncScopeID(), Agent);
+ EXPECT_EQ(Fences[1]->getOrdering(), AtomicOrdering::SequentiallyConsistent);
+ EXPECT_EQ(Fences[1]->getSyncScopeID(), Agent);
+ EXPECT_EQ(Fences[2]->getOrdering(), AtomicOrdering::Release);
+ EXPECT_EQ(Fences[2]->getSyncScopeID(), SyncScope::System);
+ EXPECT_EQ(Fences[3]->getOrdering(), AtomicOrdering::Acquire);
+ EXPECT_EQ(Fences[3]->getSyncScopeID(),
+ M->getContext().getOrInsertSyncScopeID("workgroup"));
+}
+
+TEST_F(OpenMPIRBuilderTest, CreateFlushGPUTriple) {
+ // Without IsGPU in the config, a GPU triple still gets a fence.
+ M->setTargetTriple(Triple("nvptx64-nvidia-cuda"));
+ OpenMPIRBuilder OMPBuilder(*M);
+ OMPBuilder.initialize();
+
+ IRBuilder<> Builder(BB);
+ OMPBuilder.createFlush(OpenMPIRBuilder::LocationDescription(Builder),
+ AtomicOrdering::Acquire);
+ Builder.CreateRetVoid();
+ EXPECT_FALSE(verifyModule(*M, &errs()));
+
+ auto *FI = dyn_cast<FenceInst>(&BB->front());
+ ASSERT_NE(FI, nullptr);
+ EXPECT_EQ(FI->getOrdering(), AtomicOrdering::Acquire);
+ EXPECT_EQ(FI->getSyncScopeID(),
+ M->getContext().getOrInsertSyncScopeID("device"));
+}
+
TEST_F(OpenMPIRBuilderTest, CreateBarrier) {
OpenMPIRBuilder OMPBuilder(*M);
OMPBuilder.initialize();
diff --git a/openmp/runtime/test/flush/omp_flush_memory_order.c b/openmp/runtime/test/flush/omp_flush_memory_order.c
new file mode 100644
index 00000000000000..968b418e00a965
--- /dev/null
+++ b/openmp/runtime/test/flush/omp_flush_memory_order.c
@@ -0,0 +1,41 @@
+// RUN: %libomp-compile-and-run
+// REQUIRES: clang
+
+// Message passing between two threads using flush directives with
+// memory-order clauses, which clang lowers to __kmpc_flush_explicit. This is
+// a smoke test of that entry point: on a strongly ordered host such as x86 it
+// passes even if the fences are too weak.
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <omp.h>
+
+int main() {
+ for (int n = 0; n < 1000; n++) {
+ int x = 0, y = 0, fails = 0;
+#pragma omp parallel num_threads(2) reduction(+ : fails)
+ {
+ if (omp_get_thread_num() == 0) {
+ x = 10;
+#pragma omp flush release
+#pragma omp atomic write
+ y = 1;
+ } else {
+ int tmp = 0;
+ while (tmp == 0) {
+#pragma omp atomic read
+ tmp = y;
+ }
+#pragma omp flush acquire
+ if (x != 10)
+ fails++;
+ }
+ }
+ if (fails) {
+ printf("failed\n");
+ return EXIT_FAILURE;
+ }
+ }
+ printf("passed\n");
+ return EXIT_SUCCESS;
+}
More information about the cfe-commits
mailing list