[clang] [llvm] [openmp] [OpenMP] Honour memory-order clauses on the flush directive (PR #241295)

Julian Brown via cfe-commits cfe-commits at lists.llvm.org
Wed Nov 11 03:46:23 PST 2026


https://github.com/jtb20 created https://github.com/llvm/llvm-project/pull/241295

At present the acquire, release and acq_rel clauses on 'flush' are accepted but have no effect.  Flushes are emitted as __kmpc_flush runtime calls, which implement a seq_cst fence.

AMD have had a downstream patch for a while (by Carlo Bertolli) which adds per-ordering __kmpc_flush_* entry points to the device runtime. This patch takes a slightly different approach: for the host, a seq_cst flush still calls __kmpc_flush, and a flush with any other ordering calls the __kmpc_flush_explicit entry point from #133325, so it needs a runtime that has that.  For the GPU, it emits a fence directly.

This has mild benefits: OpenMPOpt already handles fences when it converts kernels to SPMD mode, but has no knowledge of the new __kmpc_flush calls (either _explicit or Carlo's per-ordering flush calls).  Scope names differ per-GPU: they are determined using the mechanism introduced by

The lowering is done in OpenMPIRBuilder.  The createFlush function now takes an ordering and a scope, and is now used for every GPU flush. The OpenMPIRBuilder-disabled path is unaffected.  Flang/MLIR call createFlush with the defaults, so their host output is unchanged, but their GPU flushes, including those implied by atomics, now become fences instead of __kmpc_flush calls.

Since e8e05de08bea ("[OPENMP50]Add codegen for acq_rel clause in atomic|flush directives"), a flush with neither a list nor a memory-order clause has been passed to emitFlush as acq_rel. From OpenMP 5.1 it should behave as seq_cst instead, so it is now emitted as that.

Flushes implied by 'atomic' are also emitted as seq_cst.


Assisted-By: Claude

>From 9cf423acb97ed94a071cce9b524cc54c73f54444 Mon Sep 17 00:00:00 2001
From: Julian Brown <julian.brown at amd.com>
Date: Wed, 4 Nov 2026 06:59:20 -0600
Subject: [PATCH] [OpenMP] Honour memory-order clauses on the flush directive

At present the acquire, release and acq_rel clauses on 'flush' are
accepted but have no effect.  Flushes are emitted as __kmpc_flush runtime
calls, which implement a seq_cst fence.

AMD have had a downstream patch for a while (by Carlo Bertolli) which
adds per-ordering __kmpc_flush_* entry points to the device runtime.
This patch takes a slightly different approach: for the host, a seq_cst
flush still calls __kmpc_flush, and a flush with any other ordering
calls the __kmpc_flush_explicit entry point from #133325, so it needs
a runtime that has that.  For the GPU, it emits a fence directly.

This has mild benefits: OpenMPOpt already handles fences when it converts
kernels to SPMD mode, but has no knowledge of the new __kmpc_flush calls
(either _explicit or Carlo's per-ordering flush calls).  Scope names
differ per-GPU: they are determined using the mechanism introduced by

The lowering is done in OpenMPIRBuilder.  The createFlush function now
takes an ordering and a scope, and is now used for every GPU flush.
The OpenMPIRBuilder-disabled path is unaffected.  Flang/MLIR call
createFlush with the defaults, so their host output is unchanged, but
their GPU flushes, including those implied by atomics, now become fences
instead of __kmpc_flush calls.

Since e8e05de08bea ("[OPENMP50]Add codegen for acq_rel clause in
atomic|flush directives"), a flush with neither a list nor a memory-order
clause has been passed to emitFlush as acq_rel. From OpenMP 5.1 it should
behave as seq_cst instead, so it is now emitted as that.

Flushes implied by 'atomic' are also emitted as seq_cst.

Co-authored-by: Carlo Bertolli <Carlo.Bertolli at amd.com>
Assisted-By: Claude
---
 clang/docs/ReleaseNotes.md                    |  4 +
 clang/lib/CodeGen/CGOpenMPRuntime.cpp         | 30 +++++-
 clang/lib/CodeGen/CGOpenMPRuntime.h           |  2 +
 clang/lib/CodeGen/CGStmtOpenMP.cpp            | 37 ++++---
 clang/test/OpenMP/flush_codegen.cpp           | 12 +--
 .../OpenMP/flush_memory_order_codegen.cpp     | 92 ++++++++++++++++++
 .../llvm/Frontend/OpenMP/OMPIRBuilder.h       | 19 +++-
 .../include/llvm/Frontend/OpenMP/OMPKinds.def |  3 +
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     | 97 ++++++++++---------
 llvm/test/Transforms/OpenMP/add_attributes.ll |  8 ++
 .../Frontend/OpenMPIRBuilderTest.cpp          | 89 +++++++++++++++++
 .../test/flush/omp_flush_memory_order.c       | 41 ++++++++
 12 files changed, 356 insertions(+), 78 deletions(-)
 create mode 100644 clang/test/OpenMP/flush_memory_order_codegen.cpp
 create mode 100644 openmp/runtime/test/flush/omp_flush_memory_order.c

diff --git a/clang/docs/ReleaseNotes.md b/clang/docs/ReleaseNotes.md
index 0f9e7e01636d71..c0e602b72bd0d4 100644
--- a/clang/docs/ReleaseNotes.md
+++ b/clang/docs/ReleaseNotes.md
@@ -1588,6 +1588,10 @@ The `alpha.cplusplus.UseAfterLifetimeEnd` checker was renamed to `alpha.core.Use
   - A loop transformation (`tile`, `unroll`, `interchange`, ...) that consumes
     another tile's intra-tile loop.
 
+- The `acquire`, `release` and `acq_rel` clauses of the `flush` directive now
+  take effect. On GPUs a flush is emitted as a fence of the requested ordering.
+  On the host, a flush other than a seq_cst flush needs an OpenMP runtime from
+  this release.
 - Added parsing and semantic support for `dims` modifier in `num_teams`,
   `thread_limit` and `num_threads` clauses for OpenMP 6.1 or later.
 - Map-type-modifying modifiers applied to a list item with a user-defined mapper
diff --git a/clang/lib/CodeGen/CGOpenMPRuntime.cpp b/clang/lib/CodeGen/CGOpenMPRuntime.cpp
index 62a537c885fc31..9e6d3e0c5a5199 100644
--- a/clang/lib/CodeGen/CGOpenMPRuntime.cpp
+++ b/clang/lib/CodeGen/CGOpenMPRuntime.cpp
@@ -2953,16 +2953,36 @@ void CGOpenMPRuntime::emitProcBindClause(CodeGenFunction &CGF,
 
 void CGOpenMPRuntime::emitFlush(CodeGenFunction &CGF, ArrayRef<const Expr *>,
                                 SourceLocation Loc, llvm::AtomicOrdering AO) {
-  if (CGF.CGM.getLangOpts().OpenMPIRBuilder) {
-    OMPBuilder.createFlush(CGF.Builder);
-  } else {
-    if (!CGF.HaveInsertPoint())
-      return;
+  // The OpenMPIRBuilder emits a fence on a GPU, so GPUs always use it.
+  if (CGF.CGM.getLangOpts().OpenMPIRBuilder || isGPU()) {
+    OMPBuilder.createFlush(CGF.Builder, AO);
+    return;
+  }
+  if (!CGF.HaveInsertPoint())
+    return;
+  // This is the host lowering of OpenMPIRBuilder::emitFlush, but with the
+  // source location from emitUpdateLocation, as for clang's other host
+  // runtime calls.
+  // A flush with a list is a strong flush, which is treated as seq_cst.
+  if (AO == llvm::AtomicOrdering::NotAtomic)
+    AO = llvm::AtomicOrdering::SequentiallyConsistent;
+  if (AO == llvm::AtomicOrdering::SequentiallyConsistent) {
     // Build call void __kmpc_flush(ident_t *loc)
     CGF.EmitRuntimeCall(OMPBuilder.getOrCreateRuntimeFunction(
                             CGM.getModule(), OMPRTL___kmpc_flush),
                         emitUpdateLocation(CGF, Loc));
+    return;
   }
+  // Build call void __kmpc_flush_explicit(ident_t *loc, kmp_int32 order,
+  // kmp_int32 scope), where order is an __ATOMIC_* value and scope is the
+  // __MEMORY_SCOPE_* value for device scope.
+  llvm::Value *Args[] = {
+      emitUpdateLocation(CGF, Loc),
+      CGF.Builder.getInt32(static_cast<unsigned>(llvm::toCABI(AO))),
+      CGF.Builder.getInt32(static_cast<unsigned>(llvm::AtomicScope::Device))};
+  CGF.EmitRuntimeCall(OMPBuilder.getOrCreateRuntimeFunction(
+                          CGM.getModule(), OMPRTL___kmpc_flush_explicit),
+                      Args);
 }
 
 namespace {
diff --git a/clang/lib/CodeGen/CGOpenMPRuntime.h b/clang/lib/CodeGen/CGOpenMPRuntime.h
index ae55295cadc14f..2f719aafccc9cc 100644
--- a/clang/lib/CodeGen/CGOpenMPRuntime.h
+++ b/clang/lib/CodeGen/CGOpenMPRuntime.h
@@ -1141,6 +1141,8 @@ class CGOpenMPRuntime {
 
   /// Emit flush of the variables specified in 'omp flush' directive.
   /// \param Vars List of variables to flush.
+  /// \param AO Memory ordering of the flush, or NotAtomic for a flush with a
+  /// list (a strong flush).
   virtual void emitFlush(CodeGenFunction &CGF, ArrayRef<const Expr *> Vars,
                          SourceLocation Loc, llvm::AtomicOrdering AO);
 
diff --git a/clang/lib/CodeGen/CGStmtOpenMP.cpp b/clang/lib/CodeGen/CGStmtOpenMP.cpp
index 62fa820532fbd9..6959fe4ada619a 100644
--- a/clang/lib/CodeGen/CGStmtOpenMP.cpp
+++ b/clang/lib/CodeGen/CGStmtOpenMP.cpp
@@ -6262,9 +6262,17 @@ void CodeGenFunction::EmitOMPTaskgroupDirective(
 }
 
 void CodeGenFunction::EmitOMPFlushDirective(const OMPFlushDirective &S) {
-  llvm::AtomicOrdering AO = S.getSingleClause<OMPFlushClause>()
-                                ? llvm::AtomicOrdering::NotAtomic
-                                : llvm::AtomicOrdering::AcquireRelease;
+  // A flush with a list is a strong flush. Without a list the flush behaves as
+  // if the memory-order clause is seq_cst unless another one is given.
+  llvm::AtomicOrdering AO = llvm::AtomicOrdering::SequentiallyConsistent;
+  if (S.getSingleClause<OMPFlushClause>())
+    AO = llvm::AtomicOrdering::NotAtomic;
+  else if (S.getSingleClause<OMPAcqRelClause>())
+    AO = llvm::AtomicOrdering::AcquireRelease;
+  else if (S.getSingleClause<OMPAcquireClause>())
+    AO = llvm::AtomicOrdering::Acquire;
+  else if (S.getSingleClause<OMPReleaseClause>())
+    AO = llvm::AtomicOrdering::Release;
   CGM.getOpenMPRuntime().emitFlush(
       *this,
       [&S]() -> ArrayRef<const Expr *> {
@@ -6975,12 +6983,15 @@ static void emitOMPAtomicReadExpr(CodeGenFunction &CGF, llvm::AtomicOrdering AO,
   // If the read or capture clause is specified and the acquire, acq_rel, or
   // seq_cst clause is specified then the strong flush on exit from the atomic
   // operation is also an acquire flush.
+  // The flushes implied by an atomic construct are all emitted after the
+  // atomic operation, where a release flush would order nothing, so they are
+  // emitted as seq_cst flushes.
   switch (AO) {
   case llvm::AtomicOrdering::Acquire:
   case llvm::AtomicOrdering::AcquireRelease:
   case llvm::AtomicOrdering::SequentiallyConsistent:
-    CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
-                                         llvm::AtomicOrdering::Acquire);
+    CGF.CGM.getOpenMPRuntime().emitFlush(
+        CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
     break;
   case llvm::AtomicOrdering::Monotonic:
   case llvm::AtomicOrdering::Release:
@@ -7008,8 +7019,8 @@ static void emitOMPAtomicWriteExpr(CodeGenFunction &CGF,
   case llvm::AtomicOrdering::Release:
   case llvm::AtomicOrdering::AcquireRelease:
   case llvm::AtomicOrdering::SequentiallyConsistent:
-    CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
-                                         llvm::AtomicOrdering::Release);
+    CGF.CGM.getOpenMPRuntime().emitFlush(
+        CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
     break;
   case llvm::AtomicOrdering::Acquire:
   case llvm::AtomicOrdering::Monotonic:
@@ -7198,8 +7209,8 @@ static void emitOMPAtomicUpdateExpr(CodeGenFunction &CGF,
   case llvm::AtomicOrdering::Release:
   case llvm::AtomicOrdering::AcquireRelease:
   case llvm::AtomicOrdering::SequentiallyConsistent:
-    CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
-                                         llvm::AtomicOrdering::Release);
+    CGF.CGM.getOpenMPRuntime().emitFlush(
+        CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
     break;
   case llvm::AtomicOrdering::Acquire:
   case llvm::AtomicOrdering::Monotonic:
@@ -7313,17 +7324,11 @@ static void emitOMPAtomicCaptureExpr(CodeGenFunction &CGF,
     // operation is also an acquire flush.
     switch (AO) {
     case llvm::AtomicOrdering::Release:
-      CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
-                                           llvm::AtomicOrdering::Release);
-      break;
     case llvm::AtomicOrdering::Acquire:
-      CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
-                                           llvm::AtomicOrdering::Acquire);
-      break;
     case llvm::AtomicOrdering::AcquireRelease:
     case llvm::AtomicOrdering::SequentiallyConsistent:
       CGF.CGM.getOpenMPRuntime().emitFlush(
-          CGF, {}, Loc, llvm::AtomicOrdering::AcquireRelease);
+          CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
       break;
     case llvm::AtomicOrdering::Monotonic:
       break;
diff --git a/clang/test/OpenMP/flush_codegen.cpp b/clang/test/OpenMP/flush_codegen.cpp
index 022bf856d82c91..195b0186b847f0 100644
--- a/clang/test/OpenMP/flush_codegen.cpp
+++ b/clang/test/OpenMP/flush_codegen.cpp
@@ -48,9 +48,9 @@ int main() {
 #pragma omp flush(a)
   // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
   // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
-  // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
-  // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
-  // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
+  // CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 4, i32 1)
+  // CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 2, i32 1)
+  // CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 3, i32 1)
   // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
   return tmain(a);
   // CHECK: call {{.*}} [[TMAIN:@.+]](
@@ -60,9 +60,9 @@ int main() {
 // CHECK: [[TMAIN]]
 // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
 // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
-// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
-// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
-// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
+// CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 4, i32 1)
+// CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 2, i32 1)
+// CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 3, i32 1)
 // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
 // CHECK: ret
 
diff --git a/clang/test/OpenMP/flush_memory_order_codegen.cpp b/clang/test/OpenMP/flush_memory_order_codegen.cpp
new file mode 100644
index 00000000000000..64357fe15a16ea
--- /dev/null
+++ b/clang/test/OpenMP/flush_memory_order_codegen.cpp
@@ -0,0 +1,92 @@
+// Check that the memory-order clauses of 'flush' reach the host runtime and
+// become fences on GPUs, and that the flushes implied by 'atomic' stay seq_cst.
+
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -o - | FileCheck %s --check-prefix=HOST
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -fopenmp-enable-irbuilder -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -o - | FileCheck %s --check-prefix=HOST
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm-bc %s -o %t-host.bc
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host.bc -o - | FileCheck %s --check-prefixes=DEV,AMDGPU
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -fopenmp-enable-irbuilder -x c++ -triple amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host.bc -o - | FileCheck %s --check-prefixes=DEV,AMDGPU
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=nvptx64-nvidia-cuda -emit-llvm-bc %s -o %t-host-nv.bc
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple nvptx64-nvidia-cuda -fopenmp-targets=nvptx64-nvidia-cuda -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host-nv.bc -o - | FileCheck %s --check-prefixes=DEV,NVPTX
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=spirv64-intel -emit-llvm-bc %s -o %t-host-spirv.bc
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple spirv64-intel -fopenmp-targets=spirv64-intel -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host-spirv.bc -o - | FileCheck %s --check-prefixes=DEV,SPIRV
+// expected-no-diagnostics
+
+int x, v, e, d;
+
+void foo() {
+#pragma omp target
+  {
+#pragma omp flush
+#pragma omp flush(x)
+#pragma omp flush seq_cst
+#pragma omp flush acq_rel
+#pragma omp flush acquire
+#pragma omp flush release
+#pragma omp atomic read acquire
+    v = x;
+#pragma omp atomic write release
+    x = v;
+#pragma omp atomic compare seq_cst
+    if (x == e)
+      x = d;
+  }
+}
+
+// The host keeps calling __kmpc_flush for a seq_cst flush, and passes any
+// other ordering to __kmpc_flush_explicit as an __ATOMIC_* value, with the
+// device scope as a __MEMORY_SCOPE_* value.
+// HOST-LABEL: define {{.*}}@{{.*}}foo{{.*}}(
+// HOST:      call void @__kmpc_flush(ptr
+// HOST-NEXT: call void @__kmpc_flush(ptr
+// HOST-NEXT: call void @__kmpc_flush(ptr
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 4, i32 1)
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 2, i32 1)
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 3, i32 1)
+// HOST:      load atomic i32, ptr {{.*}} acquire
+// HOST-NEXT: call void @__kmpc_flush(ptr
+// HOST:      store atomic i32 {{.*}} release
+// HOST-NEXT: call void @__kmpc_flush(ptr
+// HOST:      cmpxchg ptr {{.*}} seq_cst seq_cst
+// HOST:      call void @__kmpc_flush(ptr
+
+// The device emits fences directly, at device scope.
+// DEV-LABEL: define {{.*}}@__omp_offloading_{{.*}}foo{{.*}}(
+// DEV-NOT:     call {{.*}}@__kmpc_flush
+// AMDGPU:      fence syncscope("agent") seq_cst
+// AMDGPU-NEXT: fence syncscope("agent") seq_cst
+// AMDGPU-NEXT: fence syncscope("agent") seq_cst
+// AMDGPU-NEXT: fence syncscope("agent") acq_rel
+// AMDGPU-NEXT: fence syncscope("agent") acquire
+// AMDGPU-NEXT: fence syncscope("agent") release
+// AMDGPU:      load atomic i32, ptr {{.*}} acquire
+// AMDGPU-NEXT: fence syncscope("agent") seq_cst
+// AMDGPU:      store atomic i32 {{.*}} release
+// AMDGPU-NEXT: fence syncscope("agent") seq_cst
+// AMDGPU:      cmpxchg ptr {{.*}} seq_cst seq_cst
+// AMDGPU:      fence syncscope("agent") seq_cst
+// NVPTX:       fence syncscope("device") seq_cst
+// NVPTX-NEXT:  fence syncscope("device") seq_cst
+// NVPTX-NEXT:  fence syncscope("device") seq_cst
+// NVPTX-NEXT:  fence syncscope("device") acq_rel
+// NVPTX-NEXT:  fence syncscope("device") acquire
+// NVPTX-NEXT:  fence syncscope("device") release
+// NVPTX:       load atomic i32, ptr {{.*}} acquire
+// NVPTX-NEXT:  fence syncscope("device") seq_cst
+// NVPTX:       store atomic i32 {{.*}} release
+// NVPTX-NEXT:  fence syncscope("device") seq_cst
+// NVPTX:       cmpxchg ptr {{.*}} seq_cst seq_cst
+// NVPTX:       fence syncscope("device") seq_cst
+// SPIRV:       fence syncscope("device") seq_cst
+// SPIRV-NEXT:  fence syncscope("device") seq_cst
+// SPIRV-NEXT:  fence syncscope("device") seq_cst
+// SPIRV-NEXT:  fence syncscope("device") acq_rel
+// SPIRV-NEXT:  fence syncscope("device") acquire
+// SPIRV-NEXT:  fence syncscope("device") release
+// SPIRV:       load atomic i32, ptr {{.*}} acquire
+// SPIRV-NEXT:  fence syncscope("device") seq_cst
+// SPIRV:       store atomic i32 {{.*}} release
+// SPIRV-NEXT:  fence syncscope("device") seq_cst
+// SPIRV:       cmpxchg ptr {{.*}} seq_cst seq_cst
+// SPIRV:       fence syncscope("device") seq_cst
+// DEV-NOT:     call {{.*}}@__kmpc_flush
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
index ee83c02005cb9d..16fa4357b3b57f 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
@@ -28,6 +28,7 @@
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/VirtualFileSystemFwd.h"
+#include "llvm/TargetParser/AtomicScope.h"
 #include "llvm/TargetParser/Triple.h"
 #include <forward_list>
 #include <map>
@@ -1595,7 +1596,13 @@ class OpenMPIRBuilder {
   /// Generator for '#omp flush'
   ///
   /// \param Loc The location where the flush directive was encountered
-  LLVM_ABI void createFlush(const LocationDescription &Loc);
+  /// \param AO The memory ordering of the flush. NotAtomic is a flush with a
+  /// list, which is treated as seq_cst.
+  /// \param Scope The memory scope of the flush.
+  LLVM_ABI void
+  createFlush(const LocationDescription &Loc,
+              AtomicOrdering AO = AtomicOrdering::SequentiallyConsistent,
+              AtomicScope Scope = AtomicScope::Device);
 
   /// Generate a call to the runtime to emit the diagnostic of an OpenMP
   /// `error` directive with `at(execution)`.
@@ -2688,10 +2695,16 @@ class OpenMPIRBuilder {
                                           Value *NumThreads, Value *HostPtr,
                                           ArrayRef<Value *> KernelArgs);
 
-  /// Generate a flush runtime call.
+  /// Generate a flush: a fence on a GPU, and a runtime call otherwise.
   ///
   /// \param Loc The location at which the request originated and is fulfilled.
-  LLVM_ABI void emitFlush(const LocationDescription &Loc);
+  /// \param AO The memory ordering of the flush. NotAtomic is a flush with a
+  /// list, which is treated as seq_cst.
+  /// \param Scope The memory scope of the flush.
+  LLVM_ABI void
+  emitFlush(const LocationDescription &Loc,
+            AtomicOrdering AO = AtomicOrdering::SequentiallyConsistent,
+            AtomicScope Scope = AtomicScope::Device);
 
   /// The finalization stack made up of finalize callbacks currently in-flight,
   /// wrapped into FinalizationInfo objects that reference also the finalization
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPKinds.def b/llvm/include/llvm/Frontend/OpenMP/OMPKinds.def
index f85379b1e80f82..3e6519f7b26a01 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPKinds.def
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPKinds.def
@@ -209,6 +209,7 @@ __OMP_RTL(__kmpc_cancel, false, Int32, IdentPtr, Int32, Int32)
 __OMP_RTL(__kmpc_cancel_barrier, false, Int32, IdentPtr, Int32)
 __OMP_RTL(__kmpc_error, false, Void, IdentPtr, Int32, Int8Ptr)
 __OMP_RTL(__kmpc_flush, false, Void, IdentPtr)
+__OMP_RTL(__kmpc_flush_explicit, false, Void, IdentPtr, Int32, Int32)
 __OMP_RTL(__kmpc_global_thread_num, false, Int32, IdentPtr)
 __OMP_RTL(__kmpc_get_hardware_thread_id_in_block, false, Int32, )
 __OMP_RTL(__kmpc_fork_call, true, Void, IdentPtr, Int32, ParallelTaskPtr)
@@ -703,6 +704,8 @@ __OMP_RTL_ATTRS(__kmpc_error, AttributeSet(), AttributeSet(),
                 ParamAttrs(AttributeSet(), SExt))
 __OMP_RTL_ATTRS(__kmpc_flush, BarrierAttrs, AttributeSet(),
                 ParamAttrs(ReadOnlyPtrAttrs))
+__OMP_RTL_ATTRS(__kmpc_flush_explicit, BarrierAttrs, AttributeSet(),
+                ParamAttrs(ReadOnlyPtrAttrs, SExt, SExt))
 __OMP_RTL_ATTRS(__kmpc_global_thread_num, GetterArgReadAttrs, SExt,
                 ParamAttrs(ReadOnlyPtrAttrs))
 __OMP_RTL_ATTRS(__kmpc_get_hardware_thread_id_in_block, GetterAttrs, ZExt,
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index 81fc07f44177ca..d765b1a3d28d74 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -2329,20 +2329,45 @@ OpenMPIRBuilder::InsertPointOrErrorTy OpenMPIRBuilder::createParallel(
   return AfterIP;
 }
 
-void OpenMPIRBuilder::emitFlush(const LocationDescription &Loc) {
-  // Build call void __kmpc_flush(ident_t *loc)
+void OpenMPIRBuilder::emitFlush(const LocationDescription &Loc,
+                                AtomicOrdering AO, AtomicScope Scope) {
+  // A flush with a list is a strong flush, which is treated as seq_cst.
+  if (AO == AtomicOrdering::NotAtomic)
+    AO = AtomicOrdering::SequentiallyConsistent;
+
+  // A flush without a list corresponds to atomic_thread_fence, so on a GPU
+  // emit the fence directly rather than calling into the device runtime.
+  if (Config.IsGPU.value_or(M.getTargetTriple().isGPU())) {
+    StringRef ScopeName =
+        getAtomicScopeIRString(M.getTargetTriple(), Scope).value_or("");
+    Builder.CreateFence(AO, M.getContext().getOrInsertSyncScopeID(ScopeName));
+    return;
+  }
+
   uint32_t SrcLocStrSize;
   Constant *SrcLocStr = getOrCreateSrcLocStr(Loc, SrcLocStrSize);
-  Value *Args[] = {getOrCreateIdent(SrcLocStr, SrcLocStrSize)};
-
-  createRuntimeFunctionCall(getOrCreateRuntimeFunctionPtr(OMPRTL___kmpc_flush),
-                            Args);
+  Value *Ident = getOrCreateIdent(SrcLocStr, SrcLocStrSize);
+  if (AO == AtomicOrdering::SequentiallyConsistent &&
+      Scope == AtomicScope::Device) {
+    // Build call void __kmpc_flush(ident_t *loc)
+    createRuntimeFunctionCall(
+        getOrCreateRuntimeFunctionPtr(OMPRTL___kmpc_flush), {Ident});
+    return;
+  }
+  // Build call void __kmpc_flush_explicit(ident_t *loc, kmp_int32 order,
+  // kmp_int32 scope), where order is an __ATOMIC_* value and scope is a
+  // __MEMORY_SCOPE_* value.
+  Value *Args[] = {Ident, Builder.getInt32(static_cast<unsigned>(toCABI(AO))),
+                   Builder.getInt32(static_cast<unsigned>(Scope))};
+  createRuntimeFunctionCall(
+      getOrCreateRuntimeFunctionPtr(OMPRTL___kmpc_flush_explicit), Args);
 }
 
-void OpenMPIRBuilder::createFlush(const LocationDescription &Loc) {
+void OpenMPIRBuilder::createFlush(const LocationDescription &Loc,
+                                  AtomicOrdering AO, AtomicScope Scope) {
   if (!updateToLocation(Loc))
     return;
-  emitFlush(Loc);
+  emitFlush(Loc, AO, Scope);
 }
 
 void OpenMPIRBuilder::createError(const LocationDescription &Loc, bool IsFatal,
@@ -11548,57 +11573,33 @@ bool OpenMPIRBuilder::checkAndEmitFlushAfterAtomic(
          "Unexpected Atomic Ordering.");
 
   bool Flush = false;
-  llvm::AtomicOrdering FlushAO = AtomicOrdering::Monotonic;
-
   switch (AK) {
   case Read:
-    if (AO == AtomicOrdering::Acquire || AO == AtomicOrdering::AcquireRelease ||
-        AO == AtomicOrdering::SequentiallyConsistent) {
-      FlushAO = AtomicOrdering::Acquire;
-      Flush = true;
-    }
+    // The acquire flush on exit.
+    Flush = AO == AtomicOrdering::Acquire ||
+            AO == AtomicOrdering::AcquireRelease ||
+            AO == AtomicOrdering::SequentiallyConsistent;
     break;
   case Write:
   case Compare:
   case Update:
-    if (AO == AtomicOrdering::Release || AO == AtomicOrdering::AcquireRelease ||
-        AO == AtomicOrdering::SequentiallyConsistent) {
-      FlushAO = AtomicOrdering::Release;
-      Flush = true;
-    }
+    // The release flush on entry.
+    Flush = AO == AtomicOrdering::Release ||
+            AO == AtomicOrdering::AcquireRelease ||
+            AO == AtomicOrdering::SequentiallyConsistent;
     break;
   case Capture:
-    switch (AO) {
-    case AtomicOrdering::Acquire:
-      FlushAO = AtomicOrdering::Acquire;
-      Flush = true;
-      break;
-    case AtomicOrdering::Release:
-      FlushAO = AtomicOrdering::Release;
-      Flush = true;
-      break;
-    case AtomicOrdering::AcquireRelease:
-    case AtomicOrdering::SequentiallyConsistent:
-      FlushAO = AtomicOrdering::AcquireRelease;
-      Flush = true;
-      break;
-    default:
-      // do nothing - leave silently.
-      break;
-    }
+    Flush = AO == AtomicOrdering::Acquire || AO == AtomicOrdering::Release ||
+            AO == AtomicOrdering::AcquireRelease ||
+            AO == AtomicOrdering::SequentiallyConsistent;
+    break;
   }
 
-  if (Flush) {
-    // Currently Flush RT call still doesn't take memory_ordering, so for when
-    // that happens, this tries to do the resolution of which atomic ordering
-    // to use with but issue the flush call
-    // TODO: pass `FlushAO` after memory ordering support is added
-    (void)FlushAO;
+  // The flush is emitted after the atomic operation, even the release flush
+  // that belongs on entry, and a release fence there would order nothing. So
+  // the flush is seq_cst whichever flush the construct implies.
+  if (Flush)
     emitFlush(Loc);
-  }
-
-  // for AO == AtomicOrdering::Monotonic and  all other case combinations
-  // do nothing
   return Flush;
 }
 
diff --git a/llvm/test/Transforms/OpenMP/add_attributes.ll b/llvm/test/Transforms/OpenMP/add_attributes.ll
index 3665da1fc17027..7011ab80a766c1 100644
--- a/llvm/test/Transforms/OpenMP/add_attributes.ll
+++ b/llvm/test/Transforms/OpenMP/add_attributes.ll
@@ -469,6 +469,8 @@ declare i32 @__kmpc_cancel_barrier(ptr, i32)
 
 declare void @__kmpc_flush(ptr)
 
+declare void @__kmpc_flush_explicit(ptr, i32, i32)
+
 declare i32 @__kmpc_global_thread_num(ptr)
 
 declare void @__kmpc_fork_call(ptr, i32, ptr, ...)
@@ -994,6 +996,9 @@ declare i32 @__tgt_target_kernel_nowait(ptr, i64, i32, i32, ptr, ptr, i32, ptr,
 ; CHECK: ; Function Attrs: convergent nounwind
 ; CHECK-NEXT: declare void @__kmpc_flush(ptr)
 
+; CHECK: ; Function Attrs: convergent nounwind
+; CHECK-NEXT: declare void @__kmpc_flush_explicit(ptr, i32, i32)
+
 ; CHECK: ; Function Attrs: nounwind
 ; CHECK-NEXT: declare i32 @__kmpc_global_thread_num(ptr)
 
@@ -2303,6 +2308,9 @@ declare i32 @__tgt_target_kernel_nowait(ptr, i64, i32, i32, ptr, ptr, i32, ptr,
 ; EXT: ; Function Attrs: convergent nounwind
 ; EXT-NEXT: declare void @__kmpc_flush(ptr)
 
+; EXT: ; Function Attrs: convergent nounwind
+; EXT-NEXT: declare void @__kmpc_flush_explicit(ptr, i32 signext, i32 signext)
+
 ; EXT: ; Function Attrs: nounwind
 ; EXT-NEXT: declare signext i32 @__kmpc_global_thread_num(ptr)
 
diff --git a/llvm/unittests/Frontend/OpenMPIRBuilderTest.cpp b/llvm/unittests/Frontend/OpenMPIRBuilderTest.cpp
index 45a244a1d1ab31..1ec0f4c232a8bc 100644
--- a/llvm/unittests/Frontend/OpenMPIRBuilderTest.cpp
+++ b/llvm/unittests/Frontend/OpenMPIRBuilderTest.cpp
@@ -370,6 +370,95 @@ static Value *findAggregateFromValue(Value *V) {
   return Aggregate;
 }
 
+TEST_F(OpenMPIRBuilderTest, CreateFlush) {
+  OpenMPIRBuilder OMPBuilder(*M);
+  OMPBuilder.initialize();
+
+  IRBuilder<> Builder(BB);
+  OpenMPIRBuilder::LocationDescription Loc(Builder);
+  // A seq_cst flush at device scope, or a flush with a list, is a call to
+  // __kmpc_flush. Anything else calls __kmpc_flush_explicit with the
+  // __ATOMIC_* ordering and the __MEMORY_SCOPE_* scope.
+  OMPBuilder.createFlush(Loc);
+  OMPBuilder.createFlush(Loc, AtomicOrdering::NotAtomic);
+  OMPBuilder.createFlush(Loc, AtomicOrdering::Acquire);
+  OMPBuilder.createFlush(Loc, AtomicOrdering::SequentiallyConsistent,
+                         AtomicScope::System);
+  Builder.CreateRetVoid();
+  EXPECT_FALSE(verifyModule(*M, &errs()));
+
+  SmallVector<CallInst *> Calls;
+  for (Instruction &I : *BB)
+    if (auto *CI = dyn_cast<CallInst>(&I))
+      Calls.push_back(CI);
+  ASSERT_EQ(Calls.size(), 4U);
+  EXPECT_EQ(Calls[0]->getCalledFunction()->getName(), "__kmpc_flush");
+  EXPECT_EQ(Calls[1]->getCalledFunction()->getName(), "__kmpc_flush");
+  for (auto [CI, Order, Scope] :
+       {std::tuple(Calls[2], 2, 1), std::tuple(Calls[3], 5, 0)}) {
+    EXPECT_EQ(CI->getCalledFunction()->getName(), "__kmpc_flush_explicit");
+    EXPECT_EQ(cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue(),
+              uint64_t(Order));
+    EXPECT_EQ(cast<ConstantInt>(CI->getArgOperand(2))->getZExtValue(),
+              uint64_t(Scope));
+  }
+}
+
+TEST_F(OpenMPIRBuilderTest, CreateFlushGPU) {
+  M->setTargetTriple(Triple("amdgcn-amd-amdhsa"));
+  OpenMPIRBuilder OMPBuilder(*M);
+  OMPBuilder.Config.IsTargetDevice = true;
+  OMPBuilder.Config.setIsGPU(true);
+  OMPBuilder.initialize();
+
+  IRBuilder<> Builder(BB);
+  OpenMPIRBuilder::LocationDescription Loc(Builder);
+  // On a GPU a flush is a fence of the given ordering and scope.
+  OMPBuilder.createFlush(Loc);
+  OMPBuilder.createFlush(Loc, AtomicOrdering::NotAtomic);
+  OMPBuilder.createFlush(Loc, AtomicOrdering::Release, AtomicScope::System);
+  OMPBuilder.createFlush(Loc, AtomicOrdering::Acquire, AtomicScope::Workgroup);
+  Builder.CreateRetVoid();
+  EXPECT_FALSE(verifyModule(*M, &errs()));
+
+  SmallVector<FenceInst *> Fences;
+  for (Instruction &I : *BB) {
+    EXPECT_FALSE(isa<CallInst>(I));
+    if (auto *FI = dyn_cast<FenceInst>(&I))
+      Fences.push_back(FI);
+  }
+  ASSERT_EQ(Fences.size(), 4U);
+  SyncScope::ID Agent = M->getContext().getOrInsertSyncScopeID("agent");
+  EXPECT_EQ(Fences[0]->getOrdering(), AtomicOrdering::SequentiallyConsistent);
+  EXPECT_EQ(Fences[0]->getSyncScopeID(), Agent);
+  EXPECT_EQ(Fences[1]->getOrdering(), AtomicOrdering::SequentiallyConsistent);
+  EXPECT_EQ(Fences[1]->getSyncScopeID(), Agent);
+  EXPECT_EQ(Fences[2]->getOrdering(), AtomicOrdering::Release);
+  EXPECT_EQ(Fences[2]->getSyncScopeID(), SyncScope::System);
+  EXPECT_EQ(Fences[3]->getOrdering(), AtomicOrdering::Acquire);
+  EXPECT_EQ(Fences[3]->getSyncScopeID(),
+            M->getContext().getOrInsertSyncScopeID("workgroup"));
+}
+
+TEST_F(OpenMPIRBuilderTest, CreateFlushGPUTriple) {
+  // Without IsGPU in the config, a GPU triple still gets a fence.
+  M->setTargetTriple(Triple("nvptx64-nvidia-cuda"));
+  OpenMPIRBuilder OMPBuilder(*M);
+  OMPBuilder.initialize();
+
+  IRBuilder<> Builder(BB);
+  OMPBuilder.createFlush(OpenMPIRBuilder::LocationDescription(Builder),
+                         AtomicOrdering::Acquire);
+  Builder.CreateRetVoid();
+  EXPECT_FALSE(verifyModule(*M, &errs()));
+
+  auto *FI = dyn_cast<FenceInst>(&BB->front());
+  ASSERT_NE(FI, nullptr);
+  EXPECT_EQ(FI->getOrdering(), AtomicOrdering::Acquire);
+  EXPECT_EQ(FI->getSyncScopeID(),
+            M->getContext().getOrInsertSyncScopeID("device"));
+}
+
 TEST_F(OpenMPIRBuilderTest, CreateBarrier) {
   OpenMPIRBuilder OMPBuilder(*M);
   OMPBuilder.initialize();
diff --git a/openmp/runtime/test/flush/omp_flush_memory_order.c b/openmp/runtime/test/flush/omp_flush_memory_order.c
new file mode 100644
index 00000000000000..968b418e00a965
--- /dev/null
+++ b/openmp/runtime/test/flush/omp_flush_memory_order.c
@@ -0,0 +1,41 @@
+// RUN: %libomp-compile-and-run
+// REQUIRES: clang
+
+// Message passing between two threads using flush directives with
+// memory-order clauses, which clang lowers to __kmpc_flush_explicit. This is
+// a smoke test of that entry point: on a strongly ordered host such as x86 it
+// passes even if the fences are too weak.
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <omp.h>
+
+int main() {
+  for (int n = 0; n < 1000; n++) {
+    int x = 0, y = 0, fails = 0;
+#pragma omp parallel num_threads(2) reduction(+ : fails)
+    {
+      if (omp_get_thread_num() == 0) {
+        x = 10;
+#pragma omp flush release
+#pragma omp atomic write
+        y = 1;
+      } else {
+        int tmp = 0;
+        while (tmp == 0) {
+#pragma omp atomic read
+          tmp = y;
+        }
+#pragma omp flush acquire
+        if (x != 10)
+          fails++;
+      }
+    }
+    if (fails) {
+      printf("failed\n");
+      return EXIT_FAILURE;
+    }
+  }
+  printf("passed\n");
+  return EXIT_SUCCESS;
+}



More information about the cfe-commits mailing list