[clang] [llvm] [openmp] [OpenMP] Honour memory-order clauses on the flush directive (PR #241295)

via cfe-commits cfe-commits at lists.llvm.org
Wed Nov 11 03:47:01 PST 2026


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-clang-codegen

Author: Julian Brown (jtb20)

<details>
<summary>Changes</summary>

At present the acquire, release and acq_rel clauses on 'flush' are accepted but have no effect.  Flushes are emitted as __kmpc_flush runtime calls, which implement a seq_cst fence.

AMD have had a downstream patch for a while (by Carlo Bertolli) which adds per-ordering __kmpc_flush_* entry points to the device runtime. This patch takes a slightly different approach: for the host, a seq_cst flush still calls __kmpc_flush, and a flush with any other ordering calls the __kmpc_flush_explicit entry point from #<!-- -->133325, so it needs a runtime that has that.  For the GPU, it emits a fence directly.

This has mild benefits: OpenMPOpt already handles fences when it converts kernels to SPMD mode, but has no knowledge of the new __kmpc_flush calls (either _explicit or Carlo's per-ordering flush calls).  Scope names differ per-GPU: they are determined using the mechanism introduced by

The lowering is done in OpenMPIRBuilder.  The createFlush function now takes an ordering and a scope, and is now used for every GPU flush. The OpenMPIRBuilder-disabled path is unaffected.  Flang/MLIR call createFlush with the defaults, so their host output is unchanged, but their GPU flushes, including those implied by atomics, now become fences instead of __kmpc_flush calls.

Since e8e05de08bea ("[OPENMP50]Add codegen for acq_rel clause in atomic|flush directives"), a flush with neither a list nor a memory-order clause has been passed to emitFlush as acq_rel. From OpenMP 5.1 it should behave as seq_cst instead, so it is now emitted as that.

Flushes implied by 'atomic' are also emitted as seq_cst.


Assisted-By: Claude

---

Patch is 29.88 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/241295.diff


12 Files Affected:

- (modified) clang/docs/ReleaseNotes.md (+4) 
- (modified) clang/lib/CodeGen/CGOpenMPRuntime.cpp (+25-5) 
- (modified) clang/lib/CodeGen/CGOpenMPRuntime.h (+2) 
- (modified) clang/lib/CodeGen/CGStmtOpenMP.cpp (+21-16) 
- (modified) clang/test/OpenMP/flush_codegen.cpp (+6-6) 
- (added) clang/test/OpenMP/flush_memory_order_codegen.cpp (+92) 
- (modified) llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h (+16-3) 
- (modified) llvm/include/llvm/Frontend/OpenMP/OMPKinds.def (+3) 
- (modified) llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp (+49-48) 
- (modified) llvm/test/Transforms/OpenMP/add_attributes.ll (+8) 
- (modified) llvm/unittests/Frontend/OpenMPIRBuilderTest.cpp (+89) 
- (added) openmp/runtime/test/flush/omp_flush_memory_order.c (+41) 


``````````diff
diff --git a/clang/docs/ReleaseNotes.md b/clang/docs/ReleaseNotes.md
index 0f9e7e01636d71..c0e602b72bd0d4 100644
--- a/clang/docs/ReleaseNotes.md
+++ b/clang/docs/ReleaseNotes.md
@@ -1588,6 +1588,10 @@ The `alpha.cplusplus.UseAfterLifetimeEnd` checker was renamed to `alpha.core.Use
   - A loop transformation (`tile`, `unroll`, `interchange`, ...) that consumes
     another tile's intra-tile loop.
 
+- The `acquire`, `release` and `acq_rel` clauses of the `flush` directive now
+  take effect. On GPUs a flush is emitted as a fence of the requested ordering.
+  On the host, a flush other than a seq_cst flush needs an OpenMP runtime from
+  this release.
 - Added parsing and semantic support for `dims` modifier in `num_teams`,
   `thread_limit` and `num_threads` clauses for OpenMP 6.1 or later.
 - Map-type-modifying modifiers applied to a list item with a user-defined mapper
diff --git a/clang/lib/CodeGen/CGOpenMPRuntime.cpp b/clang/lib/CodeGen/CGOpenMPRuntime.cpp
index 62a537c885fc31..9e6d3e0c5a5199 100644
--- a/clang/lib/CodeGen/CGOpenMPRuntime.cpp
+++ b/clang/lib/CodeGen/CGOpenMPRuntime.cpp
@@ -2953,16 +2953,36 @@ void CGOpenMPRuntime::emitProcBindClause(CodeGenFunction &CGF,
 
 void CGOpenMPRuntime::emitFlush(CodeGenFunction &CGF, ArrayRef<const Expr *>,
                                 SourceLocation Loc, llvm::AtomicOrdering AO) {
-  if (CGF.CGM.getLangOpts().OpenMPIRBuilder) {
-    OMPBuilder.createFlush(CGF.Builder);
-  } else {
-    if (!CGF.HaveInsertPoint())
-      return;
+  // The OpenMPIRBuilder emits a fence on a GPU, so GPUs always use it.
+  if (CGF.CGM.getLangOpts().OpenMPIRBuilder || isGPU()) {
+    OMPBuilder.createFlush(CGF.Builder, AO);
+    return;
+  }
+  if (!CGF.HaveInsertPoint())
+    return;
+  // This is the host lowering of OpenMPIRBuilder::emitFlush, but with the
+  // source location from emitUpdateLocation, as for clang's other host
+  // runtime calls.
+  // A flush with a list is a strong flush, which is treated as seq_cst.
+  if (AO == llvm::AtomicOrdering::NotAtomic)
+    AO = llvm::AtomicOrdering::SequentiallyConsistent;
+  if (AO == llvm::AtomicOrdering::SequentiallyConsistent) {
     // Build call void __kmpc_flush(ident_t *loc)
     CGF.EmitRuntimeCall(OMPBuilder.getOrCreateRuntimeFunction(
                             CGM.getModule(), OMPRTL___kmpc_flush),
                         emitUpdateLocation(CGF, Loc));
+    return;
   }
+  // Build call void __kmpc_flush_explicit(ident_t *loc, kmp_int32 order,
+  // kmp_int32 scope), where order is an __ATOMIC_* value and scope is the
+  // __MEMORY_SCOPE_* value for device scope.
+  llvm::Value *Args[] = {
+      emitUpdateLocation(CGF, Loc),
+      CGF.Builder.getInt32(static_cast<unsigned>(llvm::toCABI(AO))),
+      CGF.Builder.getInt32(static_cast<unsigned>(llvm::AtomicScope::Device))};
+  CGF.EmitRuntimeCall(OMPBuilder.getOrCreateRuntimeFunction(
+                          CGM.getModule(), OMPRTL___kmpc_flush_explicit),
+                      Args);
 }
 
 namespace {
diff --git a/clang/lib/CodeGen/CGOpenMPRuntime.h b/clang/lib/CodeGen/CGOpenMPRuntime.h
index ae55295cadc14f..2f719aafccc9cc 100644
--- a/clang/lib/CodeGen/CGOpenMPRuntime.h
+++ b/clang/lib/CodeGen/CGOpenMPRuntime.h
@@ -1141,6 +1141,8 @@ class CGOpenMPRuntime {
 
   /// Emit flush of the variables specified in 'omp flush' directive.
   /// \param Vars List of variables to flush.
+  /// \param AO Memory ordering of the flush, or NotAtomic for a flush with a
+  /// list (a strong flush).
   virtual void emitFlush(CodeGenFunction &CGF, ArrayRef<const Expr *> Vars,
                          SourceLocation Loc, llvm::AtomicOrdering AO);
 
diff --git a/clang/lib/CodeGen/CGStmtOpenMP.cpp b/clang/lib/CodeGen/CGStmtOpenMP.cpp
index 62fa820532fbd9..6959fe4ada619a 100644
--- a/clang/lib/CodeGen/CGStmtOpenMP.cpp
+++ b/clang/lib/CodeGen/CGStmtOpenMP.cpp
@@ -6262,9 +6262,17 @@ void CodeGenFunction::EmitOMPTaskgroupDirective(
 }
 
 void CodeGenFunction::EmitOMPFlushDirective(const OMPFlushDirective &S) {
-  llvm::AtomicOrdering AO = S.getSingleClause<OMPFlushClause>()
-                                ? llvm::AtomicOrdering::NotAtomic
-                                : llvm::AtomicOrdering::AcquireRelease;
+  // A flush with a list is a strong flush. Without a list the flush behaves as
+  // if the memory-order clause is seq_cst unless another one is given.
+  llvm::AtomicOrdering AO = llvm::AtomicOrdering::SequentiallyConsistent;
+  if (S.getSingleClause<OMPFlushClause>())
+    AO = llvm::AtomicOrdering::NotAtomic;
+  else if (S.getSingleClause<OMPAcqRelClause>())
+    AO = llvm::AtomicOrdering::AcquireRelease;
+  else if (S.getSingleClause<OMPAcquireClause>())
+    AO = llvm::AtomicOrdering::Acquire;
+  else if (S.getSingleClause<OMPReleaseClause>())
+    AO = llvm::AtomicOrdering::Release;
   CGM.getOpenMPRuntime().emitFlush(
       *this,
       [&S]() -> ArrayRef<const Expr *> {
@@ -6975,12 +6983,15 @@ static void emitOMPAtomicReadExpr(CodeGenFunction &CGF, llvm::AtomicOrdering AO,
   // If the read or capture clause is specified and the acquire, acq_rel, or
   // seq_cst clause is specified then the strong flush on exit from the atomic
   // operation is also an acquire flush.
+  // The flushes implied by an atomic construct are all emitted after the
+  // atomic operation, where a release flush would order nothing, so they are
+  // emitted as seq_cst flushes.
   switch (AO) {
   case llvm::AtomicOrdering::Acquire:
   case llvm::AtomicOrdering::AcquireRelease:
   case llvm::AtomicOrdering::SequentiallyConsistent:
-    CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
-                                         llvm::AtomicOrdering::Acquire);
+    CGF.CGM.getOpenMPRuntime().emitFlush(
+        CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
     break;
   case llvm::AtomicOrdering::Monotonic:
   case llvm::AtomicOrdering::Release:
@@ -7008,8 +7019,8 @@ static void emitOMPAtomicWriteExpr(CodeGenFunction &CGF,
   case llvm::AtomicOrdering::Release:
   case llvm::AtomicOrdering::AcquireRelease:
   case llvm::AtomicOrdering::SequentiallyConsistent:
-    CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
-                                         llvm::AtomicOrdering::Release);
+    CGF.CGM.getOpenMPRuntime().emitFlush(
+        CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
     break;
   case llvm::AtomicOrdering::Acquire:
   case llvm::AtomicOrdering::Monotonic:
@@ -7198,8 +7209,8 @@ static void emitOMPAtomicUpdateExpr(CodeGenFunction &CGF,
   case llvm::AtomicOrdering::Release:
   case llvm::AtomicOrdering::AcquireRelease:
   case llvm::AtomicOrdering::SequentiallyConsistent:
-    CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
-                                         llvm::AtomicOrdering::Release);
+    CGF.CGM.getOpenMPRuntime().emitFlush(
+        CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
     break;
   case llvm::AtomicOrdering::Acquire:
   case llvm::AtomicOrdering::Monotonic:
@@ -7313,17 +7324,11 @@ static void emitOMPAtomicCaptureExpr(CodeGenFunction &CGF,
     // operation is also an acquire flush.
     switch (AO) {
     case llvm::AtomicOrdering::Release:
-      CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
-                                           llvm::AtomicOrdering::Release);
-      break;
     case llvm::AtomicOrdering::Acquire:
-      CGF.CGM.getOpenMPRuntime().emitFlush(CGF, {}, Loc,
-                                           llvm::AtomicOrdering::Acquire);
-      break;
     case llvm::AtomicOrdering::AcquireRelease:
     case llvm::AtomicOrdering::SequentiallyConsistent:
       CGF.CGM.getOpenMPRuntime().emitFlush(
-          CGF, {}, Loc, llvm::AtomicOrdering::AcquireRelease);
+          CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
       break;
     case llvm::AtomicOrdering::Monotonic:
       break;
diff --git a/clang/test/OpenMP/flush_codegen.cpp b/clang/test/OpenMP/flush_codegen.cpp
index 022bf856d82c91..195b0186b847f0 100644
--- a/clang/test/OpenMP/flush_codegen.cpp
+++ b/clang/test/OpenMP/flush_codegen.cpp
@@ -48,9 +48,9 @@ int main() {
 #pragma omp flush(a)
   // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
   // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
-  // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
-  // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
-  // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
+  // CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 4, i32 1)
+  // CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 2, i32 1)
+  // CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 3, i32 1)
   // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
   return tmain(a);
   // CHECK: call {{.*}} [[TMAIN:@.+]](
@@ -60,9 +60,9 @@ int main() {
 // CHECK: [[TMAIN]]
 // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
 // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
-// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
-// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
-// CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
+// CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 4, i32 1)
+// CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 2, i32 1)
+// CHECK: call {{.*}}void @__kmpc_flush_explicit(ptr {{(@|%).+}}, i32 3, i32 1)
 // CHECK: call {{.*}}void @__kmpc_flush(ptr {{(@|%).+}})
 // CHECK: ret
 
diff --git a/clang/test/OpenMP/flush_memory_order_codegen.cpp b/clang/test/OpenMP/flush_memory_order_codegen.cpp
new file mode 100644
index 00000000000000..64357fe15a16ea
--- /dev/null
+++ b/clang/test/OpenMP/flush_memory_order_codegen.cpp
@@ -0,0 +1,92 @@
+// Check that the memory-order clauses of 'flush' reach the host runtime and
+// become fences on GPUs, and that the flushes implied by 'atomic' stay seq_cst.
+
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -o - | FileCheck %s --check-prefix=HOST
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -fopenmp-enable-irbuilder -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -o - | FileCheck %s --check-prefix=HOST
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm-bc %s -o %t-host.bc
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host.bc -o - | FileCheck %s --check-prefixes=DEV,AMDGPU
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -fopenmp-enable-irbuilder -x c++ -triple amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host.bc -o - | FileCheck %s --check-prefixes=DEV,AMDGPU
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=nvptx64-nvidia-cuda -emit-llvm-bc %s -o %t-host-nv.bc
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple nvptx64-nvidia-cuda -fopenmp-targets=nvptx64-nvidia-cuda -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host-nv.bc -o - | FileCheck %s --check-prefixes=DEV,NVPTX
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=spirv64-intel -emit-llvm-bc %s -o %t-host-spirv.bc
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=51 -x c++ -triple spirv64-intel -fopenmp-targets=spirv64-intel -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host-spirv.bc -o - | FileCheck %s --check-prefixes=DEV,SPIRV
+// expected-no-diagnostics
+
+int x, v, e, d;
+
+void foo() {
+#pragma omp target
+  {
+#pragma omp flush
+#pragma omp flush(x)
+#pragma omp flush seq_cst
+#pragma omp flush acq_rel
+#pragma omp flush acquire
+#pragma omp flush release
+#pragma omp atomic read acquire
+    v = x;
+#pragma omp atomic write release
+    x = v;
+#pragma omp atomic compare seq_cst
+    if (x == e)
+      x = d;
+  }
+}
+
+// The host keeps calling __kmpc_flush for a seq_cst flush, and passes any
+// other ordering to __kmpc_flush_explicit as an __ATOMIC_* value, with the
+// device scope as a __MEMORY_SCOPE_* value.
+// HOST-LABEL: define {{.*}}@{{.*}}foo{{.*}}(
+// HOST:      call void @__kmpc_flush(ptr
+// HOST-NEXT: call void @__kmpc_flush(ptr
+// HOST-NEXT: call void @__kmpc_flush(ptr
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 4, i32 1)
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 2, i32 1)
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 3, i32 1)
+// HOST:      load atomic i32, ptr {{.*}} acquire
+// HOST-NEXT: call void @__kmpc_flush(ptr
+// HOST:      store atomic i32 {{.*}} release
+// HOST-NEXT: call void @__kmpc_flush(ptr
+// HOST:      cmpxchg ptr {{.*}} seq_cst seq_cst
+// HOST:      call void @__kmpc_flush(ptr
+
+// The device emits fences directly, at device scope.
+// DEV-LABEL: define {{.*}}@__omp_offloading_{{.*}}foo{{.*}}(
+// DEV-NOT:     call {{.*}}@__kmpc_flush
+// AMDGPU:      fence syncscope("agent") seq_cst
+// AMDGPU-NEXT: fence syncscope("agent") seq_cst
+// AMDGPU-NEXT: fence syncscope("agent") seq_cst
+// AMDGPU-NEXT: fence syncscope("agent") acq_rel
+// AMDGPU-NEXT: fence syncscope("agent") acquire
+// AMDGPU-NEXT: fence syncscope("agent") release
+// AMDGPU:      load atomic i32, ptr {{.*}} acquire
+// AMDGPU-NEXT: fence syncscope("agent") seq_cst
+// AMDGPU:      store atomic i32 {{.*}} release
+// AMDGPU-NEXT: fence syncscope("agent") seq_cst
+// AMDGPU:      cmpxchg ptr {{.*}} seq_cst seq_cst
+// AMDGPU:      fence syncscope("agent") seq_cst
+// NVPTX:       fence syncscope("device") seq_cst
+// NVPTX-NEXT:  fence syncscope("device") seq_cst
+// NVPTX-NEXT:  fence syncscope("device") seq_cst
+// NVPTX-NEXT:  fence syncscope("device") acq_rel
+// NVPTX-NEXT:  fence syncscope("device") acquire
+// NVPTX-NEXT:  fence syncscope("device") release
+// NVPTX:       load atomic i32, ptr {{.*}} acquire
+// NVPTX-NEXT:  fence syncscope("device") seq_cst
+// NVPTX:       store atomic i32 {{.*}} release
+// NVPTX-NEXT:  fence syncscope("device") seq_cst
+// NVPTX:       cmpxchg ptr {{.*}} seq_cst seq_cst
+// NVPTX:       fence syncscope("device") seq_cst
+// SPIRV:       fence syncscope("device") seq_cst
+// SPIRV-NEXT:  fence syncscope("device") seq_cst
+// SPIRV-NEXT:  fence syncscope("device") seq_cst
+// SPIRV-NEXT:  fence syncscope("device") acq_rel
+// SPIRV-NEXT:  fence syncscope("device") acquire
+// SPIRV-NEXT:  fence syncscope("device") release
+// SPIRV:       load atomic i32, ptr {{.*}} acquire
+// SPIRV-NEXT:  fence syncscope("device") seq_cst
+// SPIRV:       store atomic i32 {{.*}} release
+// SPIRV-NEXT:  fence syncscope("device") seq_cst
+// SPIRV:       cmpxchg ptr {{.*}} seq_cst seq_cst
+// SPIRV:       fence syncscope("device") seq_cst
+// DEV-NOT:     call {{.*}}@__kmpc_flush
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
index ee83c02005cb9d..16fa4357b3b57f 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
@@ -28,6 +28,7 @@
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/VirtualFileSystemFwd.h"
+#include "llvm/TargetParser/AtomicScope.h"
 #include "llvm/TargetParser/Triple.h"
 #include <forward_list>
 #include <map>
@@ -1595,7 +1596,13 @@ class OpenMPIRBuilder {
   /// Generator for '#omp flush'
   ///
   /// \param Loc The location where the flush directive was encountered
-  LLVM_ABI void createFlush(const LocationDescription &Loc);
+  /// \param AO The memory ordering of the flush. NotAtomic is a flush with a
+  /// list, which is treated as seq_cst.
+  /// \param Scope The memory scope of the flush.
+  LLVM_ABI void
+  createFlush(const LocationDescription &Loc,
+              AtomicOrdering AO = AtomicOrdering::SequentiallyConsistent,
+              AtomicScope Scope = AtomicScope::Device);
 
   /// Generate a call to the runtime to emit the diagnostic of an OpenMP
   /// `error` directive with `at(execution)`.
@@ -2688,10 +2695,16 @@ class OpenMPIRBuilder {
                                           Value *NumThreads, Value *HostPtr,
                                           ArrayRef<Value *> KernelArgs);
 
-  /// Generate a flush runtime call.
+  /// Generate a flush: a fence on a GPU, and a runtime call otherwise.
   ///
   /// \param Loc The location at which the request originated and is fulfilled.
-  LLVM_ABI void emitFlush(const LocationDescription &Loc);
+  /// \param AO The memory ordering of the flush. NotAtomic is a flush with a
+  /// list, which is treated as seq_cst.
+  /// \param Scope The memory scope of the flush.
+  LLVM_ABI void
+  emitFlush(const LocationDescription &Loc,
+            AtomicOrdering AO = AtomicOrdering::SequentiallyConsistent,
+            AtomicScope Scope = AtomicScope::Device);
 
   /// The finalization stack made up of finalize callbacks currently in-flight,
   /// wrapped into FinalizationInfo objects that reference also the finalization
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPKinds.def b/llvm/include/llvm/Frontend/OpenMP/OMPKinds.def
index f85379b1e80f82..3e6519f7b26a01 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPKinds.def
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPKinds.def
@@ -209,6 +209,7 @@ __OMP_RTL(__kmpc_cancel, false, Int32, IdentPtr, Int32, Int32)
 __OMP_RTL(__kmpc_cancel_barrier, false, Int32, IdentPtr, Int32)
 __OMP_RTL(__kmpc_error, false, Void, IdentPtr, Int32, Int8Ptr)
 __OMP_RTL(__kmpc_flush, false, Void, IdentPtr)
+__OMP_RTL(__kmpc_flush_explicit, false, Void, IdentPtr, Int32, Int32)
 __OMP_RTL(__kmpc_global_thread_num, false, Int32, IdentPtr)
 __OMP_RTL(__kmpc_get_hardware_thread_id_in_block, false, Int32, )
 __OMP_RTL(__kmpc_fork_call, true, Void, IdentPtr, Int32, ParallelTaskPtr)
@@ -703,6 +704,8 @@ __OMP_RTL_ATTRS(__kmpc_error, AttributeSet(), AttributeSet(),
                 ParamAttrs(AttributeSet(), SExt))
 __OMP_RTL_ATTRS(__kmpc_flush, BarrierAttrs, AttributeSet(),
                 ParamAttrs(ReadOnlyPtrAttrs))
+__OMP_RTL_ATTRS(__kmpc_flush_explicit, BarrierAttrs, AttributeSet(),
+                ParamAttrs(ReadOnlyPtrAttrs, SExt, SExt))
 __OMP_RTL_ATTRS(__kmpc_global_thread_num, GetterArgReadAttrs, SExt,
                 ParamAttrs(ReadOnlyPtrAttrs))
 __OMP_RTL_ATTRS(__kmpc_get_hardware_thread_id_in_block, GetterAttrs, ZExt,
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index 81fc07f44177ca..d765b1a3d28d74 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -2329,20 +2329,45 @@ OpenMPIRBuilder::InsertPointOrErrorTy OpenMPIRBuilder::createParallel(
   return AfterIP;
 }
 
-void OpenMPIRBuilder::emitFlush(const LocationDescription &Loc) {
-  // Build call void __kmpc_flush(ident_t *loc)
+void OpenMPIRBuilder::emitFlush(const LocationDescription &Loc,
+                                AtomicOrdering AO, AtomicScope Scope) {
+  // A flush with a list is a strong flush, which is treated as seq_cst.
+  if (AO == AtomicOrdering::NotAtomic)
+    AO = AtomicOrdering::SequentiallyConsistent;
+
+  // A flush without a list corresponds to atomic_thread_fence, so on a GPU
+  // emit the fence directly rather than calling into the device runtime.
+  if (Config.IsGPU.value_or(M.getTargetTriple().isGPU())) {
+    StringRef ScopeName =
+        getAtomicScopeIRString(M.getTargetTriple(), Scope).value_or("");
+    Builder.CreateFence(AO, M.getContext().getOrInsertSyncScopeID(ScopeName));
+    return;
+  }
+
   uint32_t SrcLocStrSize;
   Constant *SrcLocStr = getOrCreateSrcLocStr(Loc, SrcLocStrSize);
-  Value *Args[] = {getOrCreateIdent(SrcLocStr, SrcLocStrSize)};
-
-  createRuntimeFunctionCall(getOrCreateRuntimeFunctionPtr(OMPRTL___kmpc_flush),
-                            Args);
+  Value *Ident = getOrCreateIdent(SrcLocStr, SrcLocStrSize);
+  if (AO == AtomicOrdering::SequentiallyConsistent &&
+      Scope == AtomicScope::Device) {
+    // Build call void __kmpc_flush(ident_t *loc)
+    createRuntimeFunctionCall(
+        getOrCreateRuntimeFunctionPtr(OMPRTL___kmpc_flush), {Ident});
+    return;
+  }
+  // Build call void __kmpc_flush_exp...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/241295


More information about the cfe-commits mailing list