[llvm-branch-commits] [clang] [llvm] [Clang][OpenMP] Add the OpenMP 6.0 memscope clause (PR #241296)

Julian Brown via llvm-branch-commits llvm-branch-commits at lists.llvm.org
Wed Nov 11 03:47:35 PST 2026


https://github.com/jtb20 created https://github.com/llvm/llvm-project/pull/241296

This patch adds support for the 'memscope' clause from OpenMP 6.0 on the 'flush' and 'atomic' directives. The clause selects the threads a flush or atomic synchronizes with: those on the device (the default), those in the contention group, or those on all devices.  On 'flush', a list may be specified alongside a memscope, but not together with a memory-order clause (OpenMP 6.0, "17.8.6 flush Construct").

For the host, the scope is passed to the __kmpc_flush_explicit libomp entry point as a __MEMORY_SCOPE_* value.  For a GPU, the scope maps to a target-specific syncscope for the fence.

Assisted-By: Claude

>From e89457ed59dc943654a038203fe3acd72ee3c7d6 Mon Sep 17 00:00:00 2001
From: Julian Brown <julian.brown at amd.com>
Date: Wed, 4 Nov 2026 06:59:45 -0600
Subject: [PATCH] [Clang][OpenMP] Add the OpenMP 6.0 memscope clause

This patch adds support for the 'memscope' clause from OpenMP 6.0 on
the 'flush' and 'atomic' directives. The clause selects the threads a
flush or atomic synchronizes with: those on the device (the default),
those in the contention group, or those on all devices.  On 'flush',
a list may be specified alongside a memscope, but not together with a
memory-order clause (OpenMP 6.0, "17.8.6 flush Construct").

For the host, the scope is passed to the __kmpc_flush_explicit libomp
entry point as a __MEMORY_SCOPE_* value.  For a GPU, the scope maps to a
target-specific syncscope for the fence.

Assisted-By: Claude
---
 clang/docs/OpenMPSupport.md                   |   2 +-
 clang/docs/ReleaseNotes.md                    |   6 +-
 clang/include/clang/AST/OpenMPClause.h        |  82 ++++++++++++++
 clang/include/clang/AST/RecursiveASTVisitor.h |   5 +
 clang/include/clang/Basic/OpenMPKinds.def     |   9 ++
 clang/include/clang/Basic/OpenMPKinds.h       |   7 ++
 clang/include/clang/Sema/SemaOpenMP.h         |   6 ++
 clang/lib/AST/OpenMPClause.cpp                |   8 ++
 clang/lib/AST/StmtProfile.cpp                 |   2 +
 clang/lib/Basic/OpenMPKinds.cpp               |  15 +++
 clang/lib/CodeGen/CGOpenMPRuntime.cpp         |  18 ++--
 clang/lib/CodeGen/CGOpenMPRuntime.h           |   9 +-
 clang/lib/CodeGen/CGStmtOpenMP.cpp            |  89 +++++++++------
 clang/lib/Parse/ParseOpenMP.cpp               |   1 +
 clang/lib/Sema/SemaOpenMP.cpp                 |  29 ++++-
 clang/lib/Sema/TreeTransform.h                |  21 ++++
 clang/lib/Serialization/ASTReader.cpp         |   9 ++
 clang/lib/Serialization/ASTWriter.cpp         |   6 ++
 clang/test/OpenMP/flush_memscope_codegen.cpp  | 101 ++++++++++++++++++
 clang/test/OpenMP/memscope_ast_print.cpp      |  44 ++++++++
 clang/test/OpenMP/memscope_messages.cpp       |  26 +++++
 clang/tools/libclang/CIndex.cpp               |   2 +
 llvm/include/llvm/Frontend/OpenMP/OMP.td      |   3 +
 .../llvm/Frontend/OpenMP/OMPIRBuilder.h       |  10 +-
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     |  14 +--
 .../Frontend/OpenMPIRBuilderTest.cpp          |  31 ++++++
 26 files changed, 499 insertions(+), 56 deletions(-)
 create mode 100644 clang/test/OpenMP/flush_memscope_codegen.cpp
 create mode 100644 clang/test/OpenMP/memscope_ast_print.cpp
 create mode 100644 clang/test/OpenMP/memscope_messages.cpp

diff --git a/clang/docs/OpenMPSupport.md b/clang/docs/OpenMPSupport.md
index 3e4311df9e4a18..218a62652cd145 100644
--- a/clang/docs/OpenMPSupport.md
+++ b/clang/docs/OpenMPSupport.md
@@ -309,7 +309,7 @@ implementation.
 | depth clause on loop fuse                                 | {none}`unclaimed`   | {none}`unclaimed`   |                                                                                                                                     |
 | workdistribute construct                                  |                     | {none}`in progress` | @skc7, @mjklemm                                                                                                                     |
 | task_iteration                                            | {none}`unclaimed`   | {none}`unclaimed`   |                                                                                                                                     |
-| memscope clause for atomic and flush                      | {none}`unclaimed`   | {none}`unclaimed`   |                                                                                                                                     |
+| memscope clause for atomic and flush                      | {part}`partial`     | {none}`unclaimed`   | Clang: flush, and the flushes implied by atomic (the atomic operation itself stays system-scope)                                    |
 | transparent clause (hull tasks)                           | {part}`partial`     | {none}`unclaimed`   | Clang parsing/sema [PR174646][PR174646]                                                                                             |
 | rule-based compound directives                            | {part}`In Progress` | {part}`In Progress` | kparzysz Testing for Fortran missing                                                                                                |
 | C23, C++23                                                | {none}`unclaimed`   |                     |                                                                                                                                     |
diff --git a/clang/docs/ReleaseNotes.md b/clang/docs/ReleaseNotes.md
index c0e602b72bd0d4..a1c2fb8ac0fab4 100644
--- a/clang/docs/ReleaseNotes.md
+++ b/clang/docs/ReleaseNotes.md
@@ -1589,9 +1589,9 @@ The `alpha.cplusplus.UseAfterLifetimeEnd` checker was renamed to `alpha.core.Use
     another tile's intra-tile loop.
 
 - The `acquire`, `release` and `acq_rel` clauses of the `flush` directive now
-  take effect. On GPUs a flush is emitted as a fence of the requested ordering.
-  On the host, a flush other than a seq_cst flush needs an OpenMP runtime from
-  this release.
+  take effect, and the OpenMP 6.0 `memscope` clause is supported on `flush` and
+  `atomic`. On the host, a flush other than a seq_cst flush at device scope needs
+  an OpenMP runtime from this release.
 - Added parsing and semantic support for `dims` modifier in `num_teams`,
   `thread_limit` and `num_threads` clauses for OpenMP 6.1 or later.
 - Map-type-modifying modifiers applied to a list item with a user-defined mapper
diff --git a/clang/include/clang/AST/OpenMPClause.h b/clang/include/clang/AST/OpenMPClause.h
index 3fc2e696b1c2d0..19d3379d06c4a4 100644
--- a/clang/include/clang/AST/OpenMPClause.h
+++ b/clang/include/clang/AST/OpenMPClause.h
@@ -2400,6 +2400,88 @@ class OMPSeverityClause final : public OMPClause {
   }
 };
 
+/// This represents the 'memscope' clause in the '#pragma omp atomic' and
+/// '#pragma omp flush' directives.
+///
+/// \code
+/// #pragma omp flush release memscope(all)
+/// \endcode
+/// In this example directive '#pragma omp flush' has simple 'memscope' clause
+/// with scope specifier 'all'.
+class OMPMemscopeClause final : public OMPClause {
+  friend class OMPClauseReader;
+
+  /// Location of '('
+  SourceLocation LParenLoc;
+
+  /// A kind of the 'memscope' clause.
+  OpenMPMemscopeClauseKind Kind = OMPC_MEMSCOPE_unknown;
+
+  /// Start location of the kind in source code.
+  SourceLocation KindKwLoc;
+
+  /// Set kind of the clause.
+  ///
+  /// \param K Kind of clause.
+  void setMemscopeKind(OpenMPMemscopeClauseKind K) { Kind = K; }
+
+  /// Set clause kind location.
+  ///
+  /// \param KLoc Kind location.
+  void setMemscopeKindKwLoc(SourceLocation KLoc) { KindKwLoc = KLoc; }
+
+  /// Sets the location of '('.
+  void setLParenLoc(SourceLocation Loc) { LParenLoc = Loc; }
+
+public:
+  /// Build 'memscope' clause with argument \a A ('all', 'cgroup' or
+  /// 'device').
+  ///
+  /// \param A Argument of the clause ('all', 'cgroup' or 'device').
+  /// \param ALoc Starting location of the argument.
+  /// \param StartLoc Starting location of the clause.
+  /// \param LParenLoc Location of '('.
+  /// \param EndLoc Ending location of the clause.
+  OMPMemscopeClause(OpenMPMemscopeClauseKind A, SourceLocation ALoc,
+                    SourceLocation StartLoc, SourceLocation LParenLoc,
+                    SourceLocation EndLoc)
+      : OMPClause(llvm::omp::OMPC_memscope, StartLoc, EndLoc),
+        LParenLoc(LParenLoc), Kind(A), KindKwLoc(ALoc) {}
+
+  /// Build an empty clause.
+  OMPMemscopeClause()
+      : OMPClause(llvm::omp::OMPC_memscope, SourceLocation(),
+                  SourceLocation()) {}
+
+  /// Returns the location of '('.
+  SourceLocation getLParenLoc() const { return LParenLoc; }
+
+  /// Returns kind of the clause.
+  OpenMPMemscopeClauseKind getMemscopeKind() const { return Kind; }
+
+  /// Returns location of clause kind.
+  SourceLocation getMemscopeKindKwLoc() const { return KindKwLoc; }
+
+  child_range children() {
+    return child_range(child_iterator(), child_iterator());
+  }
+
+  const_child_range children() const {
+    return const_child_range(const_child_iterator(), const_child_iterator());
+  }
+
+  child_range used_children() {
+    return child_range(child_iterator(), child_iterator());
+  }
+  const_child_range used_children() const {
+    return const_child_range(const_child_iterator(), const_child_iterator());
+  }
+
+  static bool classof(const OMPClause *T) {
+    return T->getClauseKind() == llvm::omp::OMPC_memscope;
+  }
+};
+
 /// This represents the 'message' clause in the '#pragma omp error' and the
 /// '#pragma omp parallel' directives.
 ///
diff --git a/clang/include/clang/AST/RecursiveASTVisitor.h b/clang/include/clang/AST/RecursiveASTVisitor.h
index 56aa9b772b1e75..39aa3dbc5b9816 100644
--- a/clang/include/clang/AST/RecursiveASTVisitor.h
+++ b/clang/include/clang/AST/RecursiveASTVisitor.h
@@ -3677,6 +3677,11 @@ bool RecursiveASTVisitor<Derived>::VisitOMPSeverityClause(OMPSeverityClause *) {
   return true;
 }
 
+template <typename Derived>
+bool RecursiveASTVisitor<Derived>::VisitOMPMemscopeClause(OMPMemscopeClause *) {
+  return true;
+}
+
 template <typename Derived>
 bool RecursiveASTVisitor<Derived>::VisitOMPMessageClause(OMPMessageClause *C) {
   TRY_TO(TraverseStmt(C->getMessageString()));
diff --git a/clang/include/clang/Basic/OpenMPKinds.def b/clang/include/clang/Basic/OpenMPKinds.def
index 5b5222a2ab4463..a29bf3cfb474fd 100644
--- a/clang/include/clang/Basic/OpenMPKinds.def
+++ b/clang/include/clang/Basic/OpenMPKinds.def
@@ -53,6 +53,9 @@
 #ifndef OPENMP_SEVERITY_KIND
 #define OPENMP_SEVERITY_KIND(Name)
 #endif
+#ifndef OPENMP_MEMSCOPE_KIND
+#define OPENMP_MEMSCOPE_KIND(Name)
+#endif
 #ifndef OPENMP_DEFAULTMAP_MODIFIER
 #define OPENMP_DEFAULTMAP_MODIFIER(Name)
 #endif
@@ -199,6 +202,11 @@ OPENMP_AT_KIND(execution)
 OPENMP_SEVERITY_KIND(fatal)
 OPENMP_SEVERITY_KIND(warning)
 
+// Scope specifiers for 'memscope' clause.
+OPENMP_MEMSCOPE_KIND(all)
+OPENMP_MEMSCOPE_KIND(cgroup)
+OPENMP_MEMSCOPE_KIND(device)
+
 // Map types for 'map' clause.
 OPENMP_MAP_KIND(alloc)
 OPENMP_MAP_KIND(to)
@@ -330,6 +338,7 @@ OPENMP_USE_DEVICE_PTR_FALLBACK_MODIFIER(fb_preserve)
 #undef OPENMP_ATOMIC_FAIL_MODIFIER
 #undef OPENMP_AT_KIND
 #undef OPENMP_SEVERITY_KIND
+#undef OPENMP_MEMSCOPE_KIND
 #undef OPENMP_MAP_KIND
 #undef OPENMP_MAP_MODIFIER_KIND
 #undef OPENMP_MOTION_MODIFIER_KIND
diff --git a/clang/include/clang/Basic/OpenMPKinds.h b/clang/include/clang/Basic/OpenMPKinds.h
index a10d2d4bb34473..1284f0aa659cb5 100644
--- a/clang/include/clang/Basic/OpenMPKinds.h
+++ b/clang/include/clang/Basic/OpenMPKinds.h
@@ -153,6 +153,13 @@ enum OpenMPSeverityClauseKind {
   OMPC_SEVERITY_unknown
 };
 
+/// OpenMP scope specifiers for 'memscope' clause.
+enum OpenMPMemscopeClauseKind {
+#define OPENMP_MEMSCOPE_KIND(Name) OMPC_MEMSCOPE_##Name,
+#include "clang/Basic/OpenMPKinds.def"
+  OMPC_MEMSCOPE_unknown
+};
+
 /// OpenMP device type for 'device_type' clause.
 enum OpenMPDeviceType {
 #define OPENMP_DEVICE_TYPE_KIND(Name) \
diff --git a/clang/include/clang/Sema/SemaOpenMP.h b/clang/include/clang/Sema/SemaOpenMP.h
index a8a291d02722ce..4926200e987a55 100644
--- a/clang/include/clang/Sema/SemaOpenMP.h
+++ b/clang/include/clang/Sema/SemaOpenMP.h
@@ -1200,6 +1200,12 @@ class SemaOpenMP : public SemaBase {
                                        SourceLocation StartLoc,
                                        SourceLocation LParenLoc,
                                        SourceLocation EndLoc);
+  /// Called on well-formed 'memscope' clause.
+  OMPClause *ActOnOpenMPMemscopeClause(OpenMPMemscopeClauseKind Kind,
+                                       SourceLocation KindLoc,
+                                       SourceLocation StartLoc,
+                                       SourceLocation LParenLoc,
+                                       SourceLocation EndLoc);
 
   /// Called on well-formed 'message' clause.
   /// passing string for message.
diff --git a/clang/lib/AST/OpenMPClause.cpp b/clang/lib/AST/OpenMPClause.cpp
index f0102720f7d33b..89bfee5ad6c716 100644
--- a/clang/lib/AST/OpenMPClause.cpp
+++ b/clang/lib/AST/OpenMPClause.cpp
@@ -180,6 +180,7 @@ const OMPClauseWithPreInit *OMPClauseWithPreInit::get(const OMPClause *C) {
   case OMPC_self_maps:
   case OMPC_at:
   case OMPC_severity:
+  case OMPC_memscope:
   case OMPC_device_type:
   case OMPC_match:
   case OMPC_nontemporal:
@@ -285,6 +286,7 @@ const OMPClauseWithPostUpdate *OMPClauseWithPostUpdate::get(const OMPClause *C)
   case OMPC_self_maps:
   case OMPC_at:
   case OMPC_severity:
+  case OMPC_memscope:
   case OMPC_message:
   case OMPC_device_type:
   case OMPC_match:
@@ -2173,6 +2175,12 @@ void OMPClausePrinter::VisitOMPSeverityClause(OMPSeverityClause *Node) {
      << ")";
 }
 
+void OMPClausePrinter::VisitOMPMemscopeClause(OMPMemscopeClause *Node) {
+  OS << "memscope("
+     << getOpenMPSimpleClauseTypeName(OMPC_memscope, Node->getMemscopeKind())
+     << ")";
+}
+
 void OMPClausePrinter::VisitOMPMessageClause(OMPMessageClause *Node) {
   OS << "message(";
   if (Expr *E = Node->getMessageString())
diff --git a/clang/lib/AST/StmtProfile.cpp b/clang/lib/AST/StmtProfile.cpp
index 476610b9956c71..20a0508da38f22 100644
--- a/clang/lib/AST/StmtProfile.cpp
+++ b/clang/lib/AST/StmtProfile.cpp
@@ -607,6 +607,8 @@ void OMPClauseProfiler::VisitOMPAtClause(const OMPAtClause *C) {}
 
 void OMPClauseProfiler::VisitOMPSeverityClause(const OMPSeverityClause *C) {}
 
+void OMPClauseProfiler::VisitOMPMemscopeClause(const OMPMemscopeClause *C) {}
+
 void OMPClauseProfiler::VisitOMPMessageClause(const OMPMessageClause *C) {
   if (C->getMessageString())
     Profiler->VisitStmt(C->getMessageString());
diff --git a/clang/lib/Basic/OpenMPKinds.cpp b/clang/lib/Basic/OpenMPKinds.cpp
index 3d06da48b03dcf..8f19f5598826ca 100644
--- a/clang/lib/Basic/OpenMPKinds.cpp
+++ b/clang/lib/Basic/OpenMPKinds.cpp
@@ -149,6 +149,11 @@ unsigned clang::getOpenMPSimpleClauseType(OpenMPClauseKind Kind, StringRef Str,
 #define OPENMP_SEVERITY_KIND(Name) .Case(#Name, OMPC_SEVERITY_##Name)
 #include "clang/Basic/OpenMPKinds.def"
         .Default(OMPC_SEVERITY_unknown);
+  case OMPC_memscope:
+    return llvm::StringSwitch<OpenMPMemscopeClauseKind>(Str)
+#define OPENMP_MEMSCOPE_KIND(Name) .Case(#Name, OMPC_MEMSCOPE_##Name)
+#include "clang/Basic/OpenMPKinds.def"
+        .Default(OMPC_MEMSCOPE_unknown);
   case OMPC_lastprivate:
     return llvm::StringSwitch<OpenMPLastprivateModifier>(Str)
 #define OPENMP_LASTPRIVATE_KIND(Name) .Case(#Name, OMPC_LASTPRIVATE_##Name)
@@ -493,6 +498,16 @@ const char *clang::getOpenMPSimpleClauseTypeName(OpenMPClauseKind Kind,
 #include "clang/Basic/OpenMPKinds.def"
     }
     llvm_unreachable("Invalid OpenMP 'severity' clause type");
+  case OMPC_memscope:
+    switch (Type) {
+    case OMPC_MEMSCOPE_unknown:
+      return "unknown";
+#define OPENMP_MEMSCOPE_KIND(Name)                                             \
+  case OMPC_MEMSCOPE_##Name:                                                   \
+    return #Name;
+#include "clang/Basic/OpenMPKinds.def"
+    }
+    llvm_unreachable("Invalid OpenMP 'memscope' clause type");
   case OMPC_lastprivate:
     switch (Type) {
     case OMPC_LASTPRIVATE_unknown:
diff --git a/clang/lib/CodeGen/CGOpenMPRuntime.cpp b/clang/lib/CodeGen/CGOpenMPRuntime.cpp
index 9e6d3e0c5a5199..324bd0ea4ffa5d 100644
--- a/clang/lib/CodeGen/CGOpenMPRuntime.cpp
+++ b/clang/lib/CodeGen/CGOpenMPRuntime.cpp
@@ -2952,10 +2952,11 @@ void CGOpenMPRuntime::emitProcBindClause(CodeGenFunction &CGF,
 }
 
 void CGOpenMPRuntime::emitFlush(CodeGenFunction &CGF, ArrayRef<const Expr *>,
-                                SourceLocation Loc, llvm::AtomicOrdering AO) {
+                                SourceLocation Loc, llvm::AtomicOrdering AO,
+                                SyncScope Scope) {
   // The OpenMPIRBuilder emits a fence on a GPU, so GPUs always use it.
   if (CGF.CGM.getLangOpts().OpenMPIRBuilder || isGPU()) {
-    OMPBuilder.createFlush(CGF.Builder, AO);
+    OMPBuilder.createFlush(CGF.Builder, AO, getAtomicScope(Scope));
     return;
   }
   if (!CGF.HaveInsertPoint())
@@ -2966,7 +2967,8 @@ void CGOpenMPRuntime::emitFlush(CodeGenFunction &CGF, ArrayRef<const Expr *>,
   // A flush with a list is a strong flush, which is treated as seq_cst.
   if (AO == llvm::AtomicOrdering::NotAtomic)
     AO = llvm::AtomicOrdering::SequentiallyConsistent;
-  if (AO == llvm::AtomicOrdering::SequentiallyConsistent) {
+  if (AO == llvm::AtomicOrdering::SequentiallyConsistent &&
+      Scope == SyncScope::DeviceScope) {
     // Build call void __kmpc_flush(ident_t *loc)
     CGF.EmitRuntimeCall(OMPBuilder.getOrCreateRuntimeFunction(
                             CGM.getModule(), OMPRTL___kmpc_flush),
@@ -2974,12 +2976,12 @@ void CGOpenMPRuntime::emitFlush(CodeGenFunction &CGF, ArrayRef<const Expr *>,
     return;
   }
   // Build call void __kmpc_flush_explicit(ident_t *loc, kmp_int32 order,
-  // kmp_int32 scope), where order is an __ATOMIC_* value and scope is the
-  // __MEMORY_SCOPE_* value for device scope.
+  // kmp_int32 scope), where order is an __ATOMIC_* value and scope is a
+  // __MEMORY_SCOPE_* value.
   llvm::Value *Args[] = {
       emitUpdateLocation(CGF, Loc),
       CGF.Builder.getInt32(static_cast<unsigned>(llvm::toCABI(AO))),
-      CGF.Builder.getInt32(static_cast<unsigned>(llvm::AtomicScope::Device))};
+      CGF.Builder.getInt32(static_cast<unsigned>(getAtomicScope(Scope)))};
   CGF.EmitRuntimeCall(OMPBuilder.getOrCreateRuntimeFunction(
                           CGM.getModule(), OMPRTL___kmpc_flush_explicit),
                       Args);
@@ -13605,8 +13607,8 @@ Address CGOpenMPSIMDRuntime::getAddrOfArtificialThreadPrivate(
 
 void CGOpenMPSIMDRuntime::emitFlush(CodeGenFunction &CGF,
                                     ArrayRef<const Expr *> Vars,
-                                    SourceLocation Loc,
-                                    llvm::AtomicOrdering AO) {
+                                    SourceLocation Loc, llvm::AtomicOrdering AO,
+                                    SyncScope Scope) {
   llvm_unreachable("Not supported in SIMD-only mode");
 }
 
diff --git a/clang/lib/CodeGen/CGOpenMPRuntime.h b/clang/lib/CodeGen/CGOpenMPRuntime.h
index 2f719aafccc9cc..ae257f351aecc4 100644
--- a/clang/lib/CodeGen/CGOpenMPRuntime.h
+++ b/clang/lib/CodeGen/CGOpenMPRuntime.h
@@ -19,6 +19,7 @@
 #include "clang/AST/Type.h"
 #include "clang/Basic/OpenMPKinds.h"
 #include "clang/Basic/SourceLocation.h"
+#include "clang/Basic/SyncScope.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/PointerIntPair.h"
 #include "llvm/ADT/SmallPtrSet.h"
@@ -1143,8 +1144,11 @@ class CGOpenMPRuntime {
   /// \param Vars List of variables to flush.
   /// \param AO Memory ordering of the flush, or NotAtomic for a flush with a
   /// list (a strong flush).
+  /// \param Scope Memory scope of the flush, as given by the 'memscope'
+  /// clause.
   virtual void emitFlush(CodeGenFunction &CGF, ArrayRef<const Expr *> Vars,
-                         SourceLocation Loc, llvm::AtomicOrdering AO);
+                         SourceLocation Loc, llvm::AtomicOrdering AO,
+                         SyncScope Scope);
 
   /// Emit task region for the task directive. The task region is
   /// emitted in several steps:
@@ -2028,7 +2032,8 @@ class CGOpenMPSIMDRuntime final : public CGOpenMPRuntime {
   /// Emit flush of the variables specified in 'omp flush' directive.
   /// \param Vars List of variables to flush.
   void emitFlush(CodeGenFunction &CGF, ArrayRef<const Expr *> Vars,
-                 SourceLocation Loc, llvm::AtomicOrdering AO) override;
+                 SourceLocation Loc, llvm::AtomicOrdering AO,
+                 SyncScope Scope) override;
 
   /// Emit task region for the task directive. The task region is
   /// emitted in several steps:
diff --git a/clang/lib/CodeGen/CGStmtOpenMP.cpp b/clang/lib/CodeGen/CGStmtOpenMP.cpp
index 6959fe4ada619a..ee660c6cf336a4 100644
--- a/clang/lib/CodeGen/CGStmtOpenMP.cpp
+++ b/clang/lib/CodeGen/CGStmtOpenMP.cpp
@@ -6261,6 +6261,26 @@ void CodeGenFunction::EmitOMPTaskgroupDirective(
   CGM.getOpenMPRuntime().emitTaskgroupRegion(*this, CodeGen, S.getBeginLoc());
 }
 
+/// Returns the memory scope given by the 'memscope' clause of \p S, or the
+/// device scope that applies when there is no such clause.
+static SyncScope getOMPMemscope(const OMPExecutableDirective &S) {
+  if (const auto *C = S.getSingleClause<OMPMemscopeClause>()) {
+    switch (C->getMemscopeKind()) {
+    case OMPC_MEMSCOPE_all:
+      return SyncScope::SystemScope;
+    case OMPC_MEMSCOPE_cgroup:
+      // A contention group is the threads of one team, and clang maps each
+      // team to one workgroup on a GPU.
+      return SyncScope::WorkgroupScope;
+    case OMPC_MEMSCOPE_device:
+      return SyncScope::DeviceScope;
+    case OMPC_MEMSCOPE_unknown:
+      llvm_unreachable("Unexpected memscope clause kind.");
+    }
+  }
+  return SyncScope::DeviceScope;
+}
+
 void CodeGenFunction::EmitOMPFlushDirective(const OMPFlushDirective &S) {
   // A flush with a list is a strong flush. Without a list the flush behaves as
   // if the memory-order clause is seq_cst unless another one is given.
@@ -6281,7 +6301,7 @@ void CodeGenFunction::EmitOMPFlushDirective(const OMPFlushDirective &S) {
                                 FlushClause->varlist_end());
         return {};
       }(),
-      S.getBeginLoc(), AO);
+      S.getBeginLoc(), AO, getOMPMemscope(S));
 }
 
 void CodeGenFunction::EmitOMPDepobjDirective(const OMPDepobjDirective &S) {
@@ -6971,7 +6991,7 @@ void CodeGenFunction::emitOMPSimpleStore(LValue LVal, RValue RVal,
 }
 
 static void emitOMPAtomicReadExpr(CodeGenFunction &CGF, llvm::AtomicOrdering AO,
-                                  const Expr *X, const Expr *V,
+                                  SyncScope Scope, const Expr *X, const Expr *V,
                                   SourceLocation Loc) {
   // v = x;
   assert(V->isLValue() && "V of 'omp atomic read' is not lvalue");
@@ -6991,7 +7011,7 @@ static void emitOMPAtomicReadExpr(CodeGenFunction &CGF, llvm::AtomicOrdering AO,
   case llvm::AtomicOrdering::AcquireRelease:
   case llvm::AtomicOrdering::SequentiallyConsistent:
     CGF.CGM.getOpenMPRuntime().emitFlush(
-        CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
+        CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent, Scope);
     break;
   case llvm::AtomicOrdering::Monotonic:
   case llvm::AtomicOrdering::Release:
@@ -7005,8 +7025,9 @@ static void emitOMPAtomicReadExpr(CodeGenFunction &CGF, llvm::AtomicOrdering AO,
 }
 
 static void emitOMPAtomicWriteExpr(CodeGenFunction &CGF,
-                                   llvm::AtomicOrdering AO, const Expr *X,
-                                   const Expr *E, SourceLocation Loc) {
+                                   llvm::AtomicOrdering AO, SyncScope Scope,
+                                   const Expr *X, const Expr *E,
+                                   SourceLocation Loc) {
   // x = expr;
   assert(X->isLValue() && "X of 'omp atomic write' is not lvalue");
   emitSimpleAtomicStore(CGF, AO, CGF.EmitLValue(X), CGF.EmitAnyExpr(E));
@@ -7020,7 +7041,7 @@ static void emitOMPAtomicWriteExpr(CodeGenFunction &CGF,
   case llvm::AtomicOrdering::AcquireRelease:
   case llvm::AtomicOrdering::SequentiallyConsistent:
     CGF.CGM.getOpenMPRuntime().emitFlush(
-        CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
+        CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent, Scope);
     break;
   case llvm::AtomicOrdering::Acquire:
   case llvm::AtomicOrdering::Monotonic:
@@ -7174,9 +7195,10 @@ std::pair<bool, RValue> CodeGenFunction::EmitOMPAtomicSimpleUpdateExpr(
 }
 
 static void emitOMPAtomicUpdateExpr(CodeGenFunction &CGF,
-                                    llvm::AtomicOrdering AO, const Expr *X,
-                                    const Expr *E, const Expr *UE,
-                                    bool IsXLHSInRHSPart, SourceLocation Loc) {
+                                    llvm::AtomicOrdering AO, SyncScope Scope,
+                                    const Expr *X, const Expr *E,
+                                    const Expr *UE, bool IsXLHSInRHSPart,
+                                    SourceLocation Loc) {
   assert(isa<BinaryOperator>(UE->IgnoreImpCasts()) &&
          "Update expr in 'atomic update' must be a binary operator.");
   const auto *BOUE = cast<BinaryOperator>(UE->IgnoreImpCasts());
@@ -7210,7 +7232,7 @@ static void emitOMPAtomicUpdateExpr(CodeGenFunction &CGF,
   case llvm::AtomicOrdering::AcquireRelease:
   case llvm::AtomicOrdering::SequentiallyConsistent:
     CGF.CGM.getOpenMPRuntime().emitFlush(
-        CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
+        CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent, Scope);
     break;
   case llvm::AtomicOrdering::Acquire:
   case llvm::AtomicOrdering::Monotonic:
@@ -7239,7 +7261,7 @@ static RValue convertToType(CodeGenFunction &CGF, RValue Value,
 }
 
 static void emitOMPAtomicCaptureExpr(CodeGenFunction &CGF,
-                                     llvm::AtomicOrdering AO,
+                                     llvm::AtomicOrdering AO, SyncScope Scope,
                                      bool IsPostfixUpdate, const Expr *V,
                                      const Expr *X, const Expr *E,
                                      const Expr *UE, bool IsXLHSInRHSPart,
@@ -7328,7 +7350,7 @@ static void emitOMPAtomicCaptureExpr(CodeGenFunction &CGF,
     case llvm::AtomicOrdering::AcquireRelease:
     case llvm::AtomicOrdering::SequentiallyConsistent:
       CGF.CGM.getOpenMPRuntime().emitFlush(
-          CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
+          CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent, Scope);
       break;
     case llvm::AtomicOrdering::Monotonic:
       break;
@@ -7341,9 +7363,9 @@ static void emitOMPAtomicCaptureExpr(CodeGenFunction &CGF,
 
 static void emitOMPAtomicCompareExpr(
     CodeGenFunction &CGF, llvm::AtomicOrdering AO, llvm::AtomicOrdering FailAO,
-    const Expr *X, const Expr *V, const Expr *R, const Expr *E, const Expr *D,
-    const Expr *CE, bool IsXBinopExpr, bool IsPostfixUpdate, bool IsFailOnly,
-    SourceLocation Loc) {
+    SyncScope Scope, const Expr *X, const Expr *V, const Expr *R, const Expr *E,
+    const Expr *D, const Expr *CE, bool IsXBinopExpr, bool IsPostfixUpdate,
+    bool IsFailOnly, SourceLocation Loc) {
   llvm::OpenMPIRBuilder &OMPBuilder =
       CGF.CGM.getOpenMPRuntime().getOMPBuilder();
 
@@ -7413,37 +7435,39 @@ static void emitOMPAtomicCompareExpr(
     // "#pragma omp atomic compare" construct.
     CGF.Builder.restoreIP(OMPBuilder.createAtomicCompare(
         CGF.Builder, XOpVal, VOpVal, ROpVal, EVal, DVal, AO, Op, IsXBinopExpr,
-        IsPostfixUpdate, IsFailOnly));
+        IsPostfixUpdate, IsFailOnly, /*IsWeak=*/false, getAtomicScope(Scope)));
   } else
     CGF.Builder.restoreIP(OMPBuilder.createAtomicCompare(
         CGF.Builder, XOpVal, VOpVal, ROpVal, EVal, DVal, AO, Op, IsXBinopExpr,
-        IsPostfixUpdate, IsFailOnly, FailAO));
+        IsPostfixUpdate, IsFailOnly, FailAO, /*IsWeak=*/false,
+        getAtomicScope(Scope)));
 }
 
 static void emitOMPAtomicExpr(CodeGenFunction &CGF, OpenMPClauseKind Kind,
                               llvm::AtomicOrdering AO,
-                              llvm::AtomicOrdering FailAO, bool IsPostfixUpdate,
-                              const Expr *X, const Expr *V, const Expr *R,
-                              const Expr *E, const Expr *UE, const Expr *D,
-                              const Expr *CE, bool IsXLHSInRHSPart,
-                              bool IsFailOnly, SourceLocation Loc) {
+                              llvm::AtomicOrdering FailAO, SyncScope Scope,
+                              bool IsPostfixUpdate, const Expr *X,
+                              const Expr *V, const Expr *R, const Expr *E,
+                              const Expr *UE, const Expr *D, const Expr *CE,
+                              bool IsXLHSInRHSPart, bool IsFailOnly,
+                              SourceLocation Loc) {
   switch (Kind) {
   case OMPC_read:
-    emitOMPAtomicReadExpr(CGF, AO, X, V, Loc);
+    emitOMPAtomicReadExpr(CGF, AO, Scope, X, V, Loc);
     break;
   case OMPC_write:
-    emitOMPAtomicWriteExpr(CGF, AO, X, E, Loc);
+    emitOMPAtomicWriteExpr(CGF, AO, Scope, X, E, Loc);
     break;
   case OMPC_unknown:
   case OMPC_update:
-    emitOMPAtomicUpdateExpr(CGF, AO, X, E, UE, IsXLHSInRHSPart, Loc);
+    emitOMPAtomicUpdateExpr(CGF, AO, Scope, X, E, UE, IsXLHSInRHSPart, Loc);
     break;
   case OMPC_capture:
-    emitOMPAtomicCaptureExpr(CGF, AO, IsPostfixUpdate, V, X, E, UE,
+    emitOMPAtomicCaptureExpr(CGF, AO, Scope, IsPostfixUpdate, V, X, E, UE,
                              IsXLHSInRHSPart, Loc);
     break;
   case OMPC_compare: {
-    emitOMPAtomicCompareExpr(CGF, AO, FailAO, X, V, R, E, D, CE,
+    emitOMPAtomicCompareExpr(CGF, AO, FailAO, Scope, X, V, R, E, D, CE,
                              IsXLHSInRHSPart, IsPostfixUpdate, IsFailOnly, Loc);
     break;
   }
@@ -7483,7 +7507,8 @@ void CodeGenFunction::EmitOMPAtomicDirective(const OMPAtomicDirective &S) {
     if (K == OMPC_weak)
       return;
     if (K == OMPC_seq_cst || K == OMPC_acq_rel || K == OMPC_acquire ||
-        K == OMPC_release || K == OMPC_relaxed || K == OMPC_hint)
+        K == OMPC_release || K == OMPC_relaxed || K == OMPC_hint ||
+        K == OMPC_memscope)
       continue;
     Kind = K;
     KindsEncountered.insert(K);
@@ -7543,10 +7568,10 @@ void CodeGenFunction::EmitOMPAtomicDirective(const OMPAtomicDirective &S) {
 
   LexicalScope Scope(*this, S.getSourceRange());
   EmitStopPoint(S.getAssociatedStmt());
-  emitOMPAtomicExpr(*this, Kind, AO, FailAO, S.isPostfixUpdate(), S.getX(),
-                    S.getV(), S.getR(), S.getExpr(), S.getUpdateExpr(),
-                    S.getD(), S.getCondExpr(), S.isXLHSInRHSPart(),
-                    S.isFailOnly(), S.getBeginLoc());
+  emitOMPAtomicExpr(*this, Kind, AO, FailAO, getOMPMemscope(S),
+                    S.isPostfixUpdate(), S.getX(), S.getV(), S.getR(),
+                    S.getExpr(), S.getUpdateExpr(), S.getD(), S.getCondExpr(),
+                    S.isXLHSInRHSPart(), S.isFailOnly(), S.getBeginLoc());
 }
 
 static void emitCommonOMPTargetDirective(CodeGenFunction &CGF,
diff --git a/clang/lib/Parse/ParseOpenMP.cpp b/clang/lib/Parse/ParseOpenMP.cpp
index f6fab0b00b0b59..bf799cd6d48343 100644
--- a/clang/lib/Parse/ParseOpenMP.cpp
+++ b/clang/lib/Parse/ParseOpenMP.cpp
@@ -3438,6 +3438,7 @@ OMPClause *Parser::ParseOpenMPClause(OpenMPDirectiveKind DKind,
   case OMPC_atomic_default_mem_order:
   case OMPC_at:
   case OMPC_severity:
+  case OMPC_memscope:
   case OMPC_bind:
     // OpenMP [2.14.3.1, Restrictions]
     //  Only a single default clause may be specified on a parallel, task or
diff --git a/clang/lib/Sema/SemaOpenMP.cpp b/clang/lib/Sema/SemaOpenMP.cpp
index 505ebc0ab69331..8cf290fc27a7b8 100644
--- a/clang/lib/Sema/SemaOpenMP.cpp
+++ b/clang/lib/Sema/SemaOpenMP.cpp
@@ -7361,6 +7361,7 @@ StmtResult SemaOpenMP::ActOnOpenMPExecutableDirective(
       case OMPC_bind:
       case OMPC_filter:
       case OMPC_severity:
+      case OMPC_memscope:
       case OMPC_message:
         continue;
       case OMPC_allocator:
@@ -12534,7 +12535,7 @@ StmtResult SemaOpenMP::ActOnOpenMPFlushDirective(ArrayRef<OMPClause *> Clauses,
   for (OMPClause *C : Clauses) {
     if (C->getClauseKind() == OMPC_flush)
       FC = cast<OMPFlushClause>(C);
-    else
+    else if (C->getClauseKind() != OMPC_memscope)
       OrderClause = C;
   }
   llvm::omp::Version OMPVersion = getLangOpts().getOpenMPVersion();
@@ -13910,6 +13911,7 @@ StmtResult SemaOpenMP::ActOnOpenMPAtomicDirective(ArrayRef<OMPClause *> Clauses,
     }
     // The following clauses are allowed, but we don't need to do anything here.
     case OMPC_hint:
+    case OMPC_memscope:
       break;
     default:
       llvm_unreachable("unknown clause is encountered");
@@ -18769,6 +18771,7 @@ OMPClause *SemaOpenMP::ActOnOpenMPSingleExprClause(OpenMPClauseKind Kind,
   case OMPC_order:
   case OMPC_at:
   case OMPC_severity:
+  case OMPC_memscope:
   case OMPC_destroy:
   case OMPC_inclusive:
   case OMPC_exclusive:
@@ -19445,6 +19448,11 @@ OMPClause *SemaOpenMP::ActOnOpenMPSimpleClause(
         static_cast<OpenMPSeverityClauseKind>(Argument), ArgumentLoc, StartLoc,
         LParenLoc, EndLoc);
     break;
+  case OMPC_memscope:
+    Res = ActOnOpenMPMemscopeClause(
+        static_cast<OpenMPMemscopeClauseKind>(Argument), ArgumentLoc, StartLoc,
+        LParenLoc, EndLoc);
+    break;
   case OMPC_threadset:
     Res = ActOnOpenMPThreadsetClause(static_cast<OpenMPThreadsetKind>(Argument),
                                      ArgumentLoc, StartLoc, LParenLoc, EndLoc);
@@ -19818,6 +19826,22 @@ OMPClause *SemaOpenMP::ActOnOpenMPSeverityClause(OpenMPSeverityClauseKind Kind,
       OMPSeverityClause(Kind, KindKwLoc, StartLoc, LParenLoc, EndLoc);
 }
 
+OMPClause *SemaOpenMP::ActOnOpenMPMemscopeClause(OpenMPMemscopeClauseKind Kind,
+                                                 SourceLocation KindKwLoc,
+                                                 SourceLocation StartLoc,
+                                                 SourceLocation LParenLoc,
+                                                 SourceLocation EndLoc) {
+  if (Kind == OMPC_MEMSCOPE_unknown) {
+    Diag(KindKwLoc, diag::err_omp_unexpected_clause_value)
+        << getListOfPossibleValues(OMPC_memscope, /*First=*/0,
+                                   /*Last=*/OMPC_MEMSCOPE_unknown)
+        << getOpenMPClauseNameForDiag(OMPC_memscope);
+    return nullptr;
+  }
+  return new (getASTContext())
+      OMPMemscopeClause(Kind, KindKwLoc, StartLoc, LParenLoc, EndLoc);
+}
+
 OMPClause *SemaOpenMP::ActOnOpenMPMessageClause(Expr *ME,
                                                 SourceLocation StartLoc,
                                                 SourceLocation LParenLoc,
@@ -20271,6 +20295,7 @@ OMPClause *SemaOpenMP::ActOnOpenMPSingleExprWithArgClause(
   case OMPC_nontemporal:
   case OMPC_at:
   case OMPC_severity:
+  case OMPC_memscope:
   case OMPC_message:
   case OMPC_destroy:
   case OMPC_novariants:
@@ -20546,6 +20571,7 @@ OMPClause *SemaOpenMP::ActOnOpenMPClause(OpenMPClauseKind Kind,
   case OMPC_order:
   case OMPC_at:
   case OMPC_severity:
+  case OMPC_memscope:
   case OMPC_message:
   case OMPC_novariants:
   case OMPC_nocontext:
@@ -21218,6 +21244,7 @@ OMPClause *SemaOpenMP::ActOnOpenMPVarListClause(OpenMPClauseKind Kind,
   case OMPC_order:
   case OMPC_at:
   case OMPC_severity:
+  case OMPC_memscope:
   case OMPC_message:
   case OMPC_destroy:
   case OMPC_novariants:
diff --git a/clang/lib/Sema/TreeTransform.h b/clang/lib/Sema/TreeTransform.h
index 2912ed99df4617..76f942045fffa9 100644
--- a/clang/lib/Sema/TreeTransform.h
+++ b/clang/lib/Sema/TreeTransform.h
@@ -2603,6 +2603,19 @@ class TreeTransform {
                                                         LParenLoc, EndLoc);
   }
 
+  /// Build a new OpenMP 'memscope' clause.
+  ///
+  /// By default, performs semantic analysis to build the new OpenMP clause.
+  /// Subclasses may override this routine to provide different behavior.
+  OMPClause *RebuildOMPMemscopeClause(OpenMPMemscopeClauseKind Kind,
+                                      SourceLocation KwLoc,
+                                      SourceLocation StartLoc,
+                                      SourceLocation LParenLoc,
+                                      SourceLocation EndLoc) {
+    return getSema().OpenMP().ActOnOpenMPMemscopeClause(Kind, KwLoc, StartLoc,
+                                                        LParenLoc, EndLoc);
+  }
+
   /// Build a new OpenMP 'message' clause.
   ///
   /// By default, performs semantic analysis to build the new OpenMP clause.
@@ -11635,6 +11648,14 @@ TreeTransform<Derived>::TransformOMPSeverityClause(OMPSeverityClause *C) {
       C->getLParenLoc(), C->getEndLoc());
 }
 
+template <typename Derived>
+OMPClause *
+TreeTransform<Derived>::TransformOMPMemscopeClause(OMPMemscopeClause *C) {
+  return getDerived().RebuildOMPMemscopeClause(
+      C->getMemscopeKind(), C->getMemscopeKindKwLoc(), C->getBeginLoc(),
+      C->getLParenLoc(), C->getEndLoc());
+}
+
 template <typename Derived>
 OMPClause *
 TreeTransform<Derived>::TransformOMPMessageClause(OMPMessageClause *C) {
diff --git a/clang/lib/Serialization/ASTReader.cpp b/clang/lib/Serialization/ASTReader.cpp
index cd7d9f24924f82..786e05614d91e9 100644
--- a/clang/lib/Serialization/ASTReader.cpp
+++ b/clang/lib/Serialization/ASTReader.cpp
@@ -11825,6 +11825,9 @@ OMPClause *OMPClauseReader::readClause() {
   case llvm::omp::OMPC_severity:
     C = new (Context) OMPSeverityClause();
     break;
+  case llvm::omp::OMPC_memscope:
+    C = new (Context) OMPMemscopeClause();
+    break;
   case llvm::omp::OMPC_message:
     C = new (Context) OMPMessageClause();
     break;
@@ -12392,6 +12395,12 @@ void OMPClauseReader::VisitOMPSeverityClause(OMPSeverityClause *C) {
   C->setSeverityKindKwLoc(Record.readSourceLocation());
 }
 
+void OMPClauseReader::VisitOMPMemscopeClause(OMPMemscopeClause *C) {
+  C->setMemscopeKind(static_cast<OpenMPMemscopeClauseKind>(Record.readInt()));
+  C->setLParenLoc(Record.readSourceLocation());
+  C->setMemscopeKindKwLoc(Record.readSourceLocation());
+}
+
 void OMPClauseReader::VisitOMPMessageClause(OMPMessageClause *C) {
   VisitOMPClauseWithPreInit(C);
   C->setMessageString(Record.readSubExpr());
diff --git a/clang/lib/Serialization/ASTWriter.cpp b/clang/lib/Serialization/ASTWriter.cpp
index de1c2449138e58..77341537d71b11 100644
--- a/clang/lib/Serialization/ASTWriter.cpp
+++ b/clang/lib/Serialization/ASTWriter.cpp
@@ -8954,6 +8954,12 @@ void OMPClauseWriter::VisitOMPSeverityClause(OMPSeverityClause *C) {
   Record.AddSourceLocation(C->getSeverityKindKwLoc());
 }
 
+void OMPClauseWriter::VisitOMPMemscopeClause(OMPMemscopeClause *C) {
+  Record.push_back(C->getMemscopeKind());
+  Record.AddSourceLocation(C->getLParenLoc());
+  Record.AddSourceLocation(C->getMemscopeKindKwLoc());
+}
+
 void OMPClauseWriter::VisitOMPMessageClause(OMPMessageClause *C) {
   VisitOMPClauseWithPreInit(C);
   Record.AddStmt(C->getMessageString());
diff --git a/clang/test/OpenMP/flush_memscope_codegen.cpp b/clang/test/OpenMP/flush_memscope_codegen.cpp
new file mode 100644
index 00000000000000..8914557ce1ab8f
--- /dev/null
+++ b/clang/test/OpenMP/flush_memscope_codegen.cpp
@@ -0,0 +1,101 @@
+// Check that the OpenMP 6.0 memscope clause sets the scope of a flush and of
+// the flushes implied by 'atomic': a __MEMORY_SCOPE_* value passed to
+// __kmpc_flush_explicit on the host, and the sync scope of the fence on GPUs.
+
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=60 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -o - | FileCheck %s --check-prefix=HOST
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=60 -fopenmp-enable-irbuilder -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -o - | FileCheck %s --check-prefix=HOST
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=60 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm-bc %s -o %t-host.bc
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=60 -x c++ -triple amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host.bc -o - | FileCheck %s --check-prefixes=DEV,AMDGPU
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=60 -fopenmp-enable-irbuilder -x c++ -triple amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host.bc -o - | FileCheck %s --check-prefixes=DEV,AMDGPU
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=60 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=nvptx64-nvidia-cuda -emit-llvm-bc %s -o %t-host-nv.bc
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=60 -x c++ -triple nvptx64-nvidia-cuda -fopenmp-targets=nvptx64-nvidia-cuda -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host-nv.bc -o - | FileCheck %s --check-prefixes=DEV,NVPTX
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=60 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=spirv64-intel -emit-llvm-bc %s -o %t-host-spirv.bc
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=60 -x c++ -triple spirv64-intel -fopenmp-targets=spirv64-intel -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host-spirv.bc -o - | FileCheck %s --check-prefixes=DEV,SPIRV
+// expected-no-diagnostics
+
+int x, v, e, d;
+
+void foo() {
+#pragma omp target
+  {
+#pragma omp flush memscope(device)
+#pragma omp flush memscope(all)
+#pragma omp flush(x) memscope(all)
+#pragma omp flush release memscope(all)
+#pragma omp flush acquire memscope(cgroup)
+#pragma omp flush acq_rel memscope(device)
+#pragma omp atomic read acquire memscope(cgroup)
+    v = x;
+#pragma omp atomic write release memscope(all)
+    x = v;
+#pragma omp atomic update seq_cst memscope(cgroup)
+    x += v;
+#pragma omp atomic compare seq_cst memscope(all)
+    if (x == e)
+      x = d;
+  }
+}
+
+// memscope(device) is the default, so a seq_cst flush with it still calls
+// __kmpc_flush. Scopes: 0 = system (all), 1 = device, 2 = workgroup (cgroup).
+// HOST-LABEL: define {{.*}}@{{.*}}foo{{.*}}(
+// HOST:      call void @__kmpc_flush(ptr
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 5, i32 0)
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 5, i32 0)
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 3, i32 0)
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 2, i32 2)
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 4, i32 1)
+// HOST:      load atomic i32, ptr {{.*}} acquire
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 5, i32 2)
+// HOST:      store atomic i32 {{.*}} release
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 5, i32 0)
+// HOST:      atomicrmw add ptr {{.*}} seq_cst
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 5, i32 2)
+// HOST:      cmpxchg ptr {{.*}} seq_cst seq_cst
+// HOST:      call void @__kmpc_flush_explicit(ptr {{.*}}, i32 5, i32 0)
+
+// DEV-LABEL: define {{.*}}@__omp_offloading_{{.*}}foo{{.*}}(
+// DEV-NOT:     call {{.*}}@__kmpc_flush
+// AMDGPU:      fence syncscope("agent") seq_cst
+// AMDGPU-NEXT: fence seq_cst
+// AMDGPU-NEXT: fence seq_cst
+// AMDGPU-NEXT: fence release
+// AMDGPU-NEXT: fence syncscope("workgroup") acquire
+// AMDGPU-NEXT: fence syncscope("agent") acq_rel
+// AMDGPU:      load atomic i32, ptr {{.*}} acquire
+// AMDGPU-NEXT: fence syncscope("workgroup") seq_cst
+// AMDGPU:      store atomic i32 {{.*}} release
+// AMDGPU-NEXT: fence seq_cst
+// AMDGPU:      atomicrmw add ptr {{.*}} seq_cst
+// AMDGPU-NEXT: fence syncscope("workgroup") seq_cst
+// AMDGPU:      cmpxchg ptr {{.*}} seq_cst seq_cst
+// AMDGPU:      fence seq_cst
+// NVPTX:       fence syncscope("device") seq_cst
+// NVPTX-NEXT:  fence seq_cst
+// NVPTX-NEXT:  fence seq_cst
+// NVPTX-NEXT:  fence release
+// NVPTX-NEXT:  fence syncscope("block") acquire
+// NVPTX-NEXT:  fence syncscope("device") acq_rel
+// NVPTX:       load atomic i32, ptr {{.*}} acquire
+// NVPTX-NEXT:  fence syncscope("block") seq_cst
+// NVPTX:       store atomic i32 {{.*}} release
+// NVPTX-NEXT:  fence seq_cst
+// NVPTX:       atomicrmw add ptr {{.*}} seq_cst
+// NVPTX-NEXT:  fence syncscope("block") seq_cst
+// NVPTX:       cmpxchg ptr {{.*}} seq_cst seq_cst
+// NVPTX:       fence seq_cst
+// SPIRV:       fence syncscope("device") seq_cst
+// SPIRV-NEXT:  fence seq_cst
+// SPIRV-NEXT:  fence seq_cst
+// SPIRV-NEXT:  fence release
+// SPIRV-NEXT:  fence syncscope("workgroup") acquire
+// SPIRV-NEXT:  fence syncscope("device") acq_rel
+// SPIRV:       load atomic i32, ptr {{.*}} acquire
+// SPIRV-NEXT:  fence syncscope("workgroup") seq_cst
+// SPIRV:       store atomic i32 {{.*}} release
+// SPIRV-NEXT:  fence seq_cst
+// SPIRV:       atomicrmw add ptr {{.*}} seq_cst
+// SPIRV-NEXT:  fence syncscope("workgroup") seq_cst
+// SPIRV:       cmpxchg ptr {{.*}} seq_cst seq_cst
+// SPIRV:       fence seq_cst
+// DEV-NOT:     call {{.*}}@__kmpc_flush
diff --git a/clang/test/OpenMP/memscope_ast_print.cpp b/clang/test/OpenMP/memscope_ast_print.cpp
new file mode 100644
index 00000000000000..c23a45bb7a3d0d
--- /dev/null
+++ b/clang/test/OpenMP/memscope_ast_print.cpp
@@ -0,0 +1,44 @@
+// RUN: %clang_cc1 -verify -fopenmp -fopenmp-version=60 -ast-print %s | FileCheck %s
+// RUN: %clang_cc1 -fopenmp -x c++ -std=c++11 -fopenmp-version=60 -emit-pch -o %t %s
+// RUN: %clang_cc1 -fopenmp -std=c++11 -fopenmp-version=60 -include-pch %t -verify %s -ast-print | FileCheck %s
+// RUN: %clang_cc1 -fopenmp -std=c++11 -fopenmp-version=60 -include-pch %t -ast-dump-all %s | FileCheck %s --check-prefix=DUMP
+// expected-no-diagnostics
+
+#ifndef HEADER
+#define HEADER
+
+template <class T> T tmain(T x) {
+  T v;
+#pragma omp flush memscope(all)
+#pragma omp flush release memscope(cgroup)
+#pragma omp atomic read acquire memscope(device)
+  v = x;
+  return v;
+}
+
+// CHECK-LABEL: template <class T> T tmain(T x) {
+// CHECK: #pragma omp flush memscope(all)
+// CHECK-NEXT: #pragma omp flush release memscope(cgroup)
+// CHECK-NEXT: #pragma omp atomic read acquire memscope(device)
+// CHECK-NEXT: v = x;
+
+// CHECK-LABEL: template<> int tmain<int>(int x) {
+// CHECK: #pragma omp flush memscope(all)
+// CHECK-NEXT: #pragma omp flush release memscope(cgroup)
+// CHECK-NEXT: #pragma omp atomic read acquire memscope(device)
+// CHECK-NEXT: v = x;
+
+// DUMP-LABEL: FunctionDecl {{.*}} tmain 'int (int)'
+// DUMP: OMPFlushDirective
+// DUMP-NEXT: OMPMemscopeClause
+// DUMP: OMPAtomicDirective
+// DUMP-NEXT: OMPReadClause
+// DUMP-NEXT: OMPAcquireClause
+// DUMP-NEXT: OMPMemscopeClause
+
+int main() {
+  int x = 0;
+  return tmain(x);
+}
+
+#endif
diff --git a/clang/test/OpenMP/memscope_messages.cpp b/clang/test/OpenMP/memscope_messages.cpp
new file mode 100644
index 00000000000000..10aeec3d22e616
--- /dev/null
+++ b/clang/test/OpenMP/memscope_messages.cpp
@@ -0,0 +1,26 @@
+// RUN: %clang_cc1 -verify=expected,omp51 -fopenmp -fopenmp-version=51 -ferror-limit 100 %s
+// RUN: %clang_cc1 -verify=expected,omp60 -fopenmp -fopenmp-version=60 -ferror-limit 100 %s
+
+// RUN: %clang_cc1 -verify=expected,omp51 -fopenmp-simd -fopenmp-version=51 -ferror-limit 100 %s
+// RUN: %clang_cc1 -verify=expected,omp60 -fopenmp-simd -fopenmp-version=60 -ferror-limit 100 %s
+
+int x, v;
+
+void foo() {
+#pragma omp flush memscope(device) // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp flush'}}
+#pragma omp flush release memscope(all) // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp flush'}}
+#pragma omp flush memscope(cgroup) acquire // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp flush'}}
+#pragma omp flush memscope(system) // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp flush'}} omp60-error {{expected 'all', 'cgroup' or 'device' in OpenMP clause 'memscope'}}
+#pragma omp flush memscope( // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp flush'}} omp60-error {{expected 'all', 'cgroup' or 'device' in OpenMP clause 'memscope'}} expected-error {{expected ')'}} expected-note {{to match this '('}}
+#pragma omp flush memscope // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp flush'}} expected-error {{expected '(' after 'memscope'}}
+#pragma omp flush memscope(all) memscope(device) // omp51-error 2 {{unexpected OpenMP clause 'memscope' in directive '#pragma omp flush'}} expected-error {{directive '#pragma omp flush' cannot contain more than one 'memscope' clause}}
+#pragma omp flush(x) memscope(all) // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp flush'}}
+#pragma omp flush(x) release memscope(all) // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp flush'}} expected-error {{'flush' directive with memory order clause 'release' cannot have the list}} expected-note {{memory order clause 'release' is specified here}}
+#pragma omp atomic read memscope(device) // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp atomic'}}
+  v = x;
+#pragma omp atomic seq_cst memscope(all) // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp atomic'}}
+  x += v;
+#pragma omp atomic memscope(cgroup) memscope(all) // omp51-error 2 {{unexpected OpenMP clause 'memscope' in directive '#pragma omp atomic'}} expected-error {{directive '#pragma omp atomic' cannot contain more than one 'memscope' clause}}
+  x++;
+#pragma omp barrier memscope(all) // expected-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp barrier'}}
+}
diff --git a/clang/tools/libclang/CIndex.cpp b/clang/tools/libclang/CIndex.cpp
index a80a61f6492767..b1c6f2650a5490 100644
--- a/clang/tools/libclang/CIndex.cpp
+++ b/clang/tools/libclang/CIndex.cpp
@@ -2550,6 +2550,8 @@ void OMPClauseEnqueue::VisitOMPAtClause(const OMPAtClause *) {}
 
 void OMPClauseEnqueue::VisitOMPSeverityClause(const OMPSeverityClause *) {}
 
+void OMPClauseEnqueue::VisitOMPMemscopeClause(const OMPMemscopeClause *) {}
+
 void OMPClauseEnqueue::VisitOMPMessageClause(const OMPMessageClause *) {}
 
 void OMPClauseEnqueue::VisitOMPDeviceClause(const OMPDeviceClause *C) {
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMP.td b/llvm/include/llvm/Frontend/OpenMP/OMP.td
index 660ab03f9516cd..af29411f4997c5 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMP.td
+++ b/llvm/include/llvm/Frontend/OpenMP/OMP.td
@@ -339,6 +339,7 @@ def OMPC_Match : Clause<[Spelling<"match">]> {
   let flangClass = "OmpMatchClause";
 }
 def OMPC_Memscope : Clause<[Spelling<"memscope">]> {
+  let clangClass = "OMPMemscopeClause";
 }
 def OMP_MEMORY_ORDER_SeqCst : EnumVal<"seq_cst", 1, 1> {}
 def OMP_MEMORY_ORDER_AcqRel : EnumVal<"acq_rel", 2, 1> {}
@@ -716,6 +717,7 @@ def OMP_Atomic : Directive<[Spelling<"atomic">]> {
     VersionedClause<OMPC_Compare, 51>,
     VersionedClause<OMPC_Fail, 51>,
     VersionedClause<OMPC_Hint, 50>,
+    VersionedClause<OMPC_Memscope, 60>,
     VersionedClause<OMPC_Read>,
     VersionedClause<OMPC_Relaxed, 50>,
     VersionedClause<OMPC_Release, 50>,
@@ -1000,6 +1002,7 @@ def OMP_Flush : Directive<[Spelling<"flush">]> {
     // TODO This should ne `none` instead. Comment carried over from
     // OMPKinds.def.
     VersionedClause<OMPC_Flush>,
+    VersionedClause<OMPC_Memscope, 60>,
     VersionedClause<OMPC_Release, 50>,
     VersionedClause<OMPC_SeqCst, 51>,
   ];
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
index 16fa4357b3b57f..ca275213fa702d 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
@@ -4092,10 +4092,12 @@ class OpenMPIRBuilder {
   /// \param Loc    The insert and source location description.
   /// \param AO     The required atomic ordering
   /// \param AK     The OpenMP atomic operation kind used.
+  /// \param Scope  The memory scope of the flush.
   ///
   /// \returns		wether a flush was emitted or not
   bool checkAndEmitFlushAfterAtomic(const LocationDescription &Loc,
-                                    AtomicOrdering AO, AtomicKind AK);
+                                    AtomicOrdering AO, AtomicKind AK,
+                                    AtomicScope Scope = AtomicScope::Device);
 
   /// Emit atomic update for constructs: X = X BinOp Expr ,or X = Expr BinOp X
   /// For complex Operations: X = UpdateOp(X) => CmpExch X, old_X, UpdateOp(X)
@@ -4304,12 +4306,14 @@ class OpenMPIRBuilder {
       const LocationDescription &Loc, AtomicOpValue &X, AtomicOpValue &V,
       AtomicOpValue &R, Value *E, Value *D, AtomicOrdering AO,
       omp::OMPAtomicCompareOp Op, bool IsXBinopExpr, bool IsPostfixUpdate,
-      bool IsFailOnly, bool IsWeak = false);
+      bool IsFailOnly, bool IsWeak = false,
+      AtomicScope Scope = AtomicScope::Device);
   LLVM_ABI InsertPointTy createAtomicCompare(
       const LocationDescription &Loc, AtomicOpValue &X, AtomicOpValue &V,
       AtomicOpValue &R, Value *E, Value *D, AtomicOrdering AO,
       omp::OMPAtomicCompareOp Op, bool IsXBinopExpr, bool IsPostfixUpdate,
-      bool IsFailOnly, AtomicOrdering Failure, bool IsWeak = false);
+      bool IsFailOnly, AtomicOrdering Failure, bool IsWeak = false,
+      AtomicScope Scope = AtomicScope::Device);
 
   /// Create the control flow structure of a canonical OpenMP loop.
   ///
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index d765b1a3d28d74..a9350976490183 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -11567,7 +11567,8 @@ Error OpenMPIRBuilder::emitIfClause(Value *Cond, BodyGenCallbackTy ThenGen,
 }
 
 bool OpenMPIRBuilder::checkAndEmitFlushAfterAtomic(
-    const LocationDescription &Loc, llvm::AtomicOrdering AO, AtomicKind AK) {
+    const LocationDescription &Loc, llvm::AtomicOrdering AO, AtomicKind AK,
+    AtomicScope Scope) {
   assert(!(AO == AtomicOrdering::NotAtomic ||
            AO == llvm::AtomicOrdering::Unordered) &&
          "Unexpected Atomic Ordering.");
@@ -11599,7 +11600,7 @@ bool OpenMPIRBuilder::checkAndEmitFlushAfterAtomic(
   // that belongs on entry, and a release fence there would order nothing. So
   // the flush is seq_cst whichever flush the construct implies.
   if (Flush)
-    emitFlush(Loc);
+    emitFlush(Loc, AtomicOrdering::SequentiallyConsistent, Scope);
   return Flush;
 }
 
@@ -12004,18 +12005,19 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
     const LocationDescription &Loc, AtomicOpValue &X, AtomicOpValue &V,
     AtomicOpValue &R, Value *E, Value *D, AtomicOrdering AO,
     omp::OMPAtomicCompareOp Op, bool IsXBinopExpr, bool IsPostfixUpdate,
-    bool IsFailOnly, bool IsWeak) {
+    bool IsFailOnly, bool IsWeak, AtomicScope Scope) {
 
   AtomicOrdering Failure = AtomicCmpXchgInst::getStrongestFailureOrdering(AO);
   return createAtomicCompare(Loc, X, V, R, E, D, AO, Op, IsXBinopExpr,
-                             IsPostfixUpdate, IsFailOnly, Failure, IsWeak);
+                             IsPostfixUpdate, IsFailOnly, Failure, IsWeak,
+                             Scope);
 }
 
 OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
     const LocationDescription &Loc, AtomicOpValue &X, AtomicOpValue &V,
     AtomicOpValue &R, Value *E, Value *D, AtomicOrdering AO,
     omp::OMPAtomicCompareOp Op, bool IsXBinopExpr, bool IsPostfixUpdate,
-    bool IsFailOnly, AtomicOrdering Failure, bool IsWeak) {
+    bool IsFailOnly, AtomicOrdering Failure, bool IsWeak, AtomicScope Scope) {
 
   if (!updateToLocation(Loc))
     return Loc.IP;
@@ -12357,7 +12359,7 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
     }
   }
 
-  checkAndEmitFlushAfterAtomic(Loc, AO, AtomicKind::Compare);
+  checkAndEmitFlushAfterAtomic(Loc, AO, AtomicKind::Compare, Scope);
 
   return Builder.saveIP();
 }
diff --git a/llvm/unittests/Frontend/OpenMPIRBuilderTest.cpp b/llvm/unittests/Frontend/OpenMPIRBuilderTest.cpp
index 1ec0f4c232a8bc..42358bb085086a 100644
--- a/llvm/unittests/Frontend/OpenMPIRBuilderTest.cpp
+++ b/llvm/unittests/Frontend/OpenMPIRBuilderTest.cpp
@@ -4313,6 +4313,37 @@ TEST_F(OpenMPIRBuilderTest, OMPAtomicCapture) {
   EXPECT_FALSE(verifyModule(*M, &errs()));
 }
 
+TEST_F(OpenMPIRBuilderTest, OMPAtomicCompareScope) {
+  // The flush after a seq_cst atomic compare uses the given scope.
+  M->setTargetTriple(Triple("nvptx64-nvidia-cuda"));
+  OpenMPIRBuilder OMPBuilder(*M);
+  OMPBuilder.Config.IsTargetDevice = true;
+  OMPBuilder.Config.setIsGPU(true);
+  OMPBuilder.initialize();
+  IRBuilder<> Builder(BB);
+
+  LLVMContext &Ctx = M->getContext();
+  IntegerType *Int32 = Type::getInt32Ty(Ctx);
+  AllocaInst *XVal = Builder.CreateAlloca(Int32);
+  OpenMPIRBuilder::AtomicOpValue X = {XVal, Int32, true, false};
+  OpenMPIRBuilder::AtomicOpValue V = {nullptr, nullptr, false, false};
+  OpenMPIRBuilder::AtomicOpValue R = {nullptr, nullptr, false, false};
+  ConstantInt *Expr = ConstantInt::get(Int32, 1U);
+  ConstantInt *D = ConstantInt::get(Int32, 2U);
+
+  Builder.restoreIP(OMPBuilder.createAtomicCompare(
+      Builder, X, V, R, Expr, D, AtomicOrdering::SequentiallyConsistent,
+      OMPAtomicCompareOp::EQ, true, false, false, /*IsWeak=*/false,
+      AtomicScope::Workgroup));
+  Builder.CreateRetVoid();
+  EXPECT_FALSE(verifyModule(*M, &errs()));
+
+  auto *Fence = dyn_cast<FenceInst>(BB->getTerminator()->getPrevNode());
+  ASSERT_NE(Fence, nullptr);
+  EXPECT_EQ(Fence->getOrdering(), AtomicOrdering::SequentiallyConsistent);
+  EXPECT_EQ(Fence->getSyncScopeID(), Ctx.getOrInsertSyncScopeID("block"));
+}
+
 TEST_F(OpenMPIRBuilderTest, OMPAtomicCompare) {
   OpenMPIRBuilder OMPBuilder(*M);
   OMPBuilder.Config.IsTargetDevice = false;



More information about the llvm-branch-commits mailing list