[llvm-branch-commits] [clang] [llvm] [Clang][OpenMP] Add the OpenMP 6.0 memscope clause (PR #241296)
Julian Brown via llvm-branch-commits
llvm-branch-commits at lists.llvm.org
Wed Nov 11 03:47:35 PST 2026
https://github.com/jtb20 created https://github.com/llvm/llvm-project/pull/241296
This patch adds support for the 'memscope' clause from OpenMP 6.0 on the 'flush' and 'atomic' directives. The clause selects the threads a flush or atomic synchronizes with: those on the device (the default), those in the contention group, or those on all devices. On 'flush', a list may be specified alongside a memscope, but not together with a memory-order clause (OpenMP 6.0, "17.8.6 flush Construct").
For the host, the scope is passed to the __kmpc_flush_explicit libomp entry point as a __MEMORY_SCOPE_* value. For a GPU, the scope maps to a target-specific syncscope for the fence.
Assisted-By: Claude
>From e89457ed59dc943654a038203fe3acd72ee3c7d6 Mon Sep 17 00:00:00 2001
From: Julian Brown <julian.brown at amd.com>
Date: Wed, 4 Nov 2026 06:59:45 -0600
Subject: [PATCH] [Clang][OpenMP] Add the OpenMP 6.0 memscope clause
This patch adds support for the 'memscope' clause from OpenMP 6.0 on
the 'flush' and 'atomic' directives. The clause selects the threads a
flush or atomic synchronizes with: those on the device (the default),
those in the contention group, or those on all devices. On 'flush',
a list may be specified alongside a memscope, but not together with a
memory-order clause (OpenMP 6.0, "17.8.6 flush Construct").
For the host, the scope is passed to the __kmpc_flush_explicit libomp
entry point as a __MEMORY_SCOPE_* value. For a GPU, the scope maps to a
target-specific syncscope for the fence.
Assisted-By: Claude
---
clang/docs/OpenMPSupport.md | 2 +-
clang/docs/ReleaseNotes.md | 6 +-
clang/include/clang/AST/OpenMPClause.h | 82 ++++++++++++++
clang/include/clang/AST/RecursiveASTVisitor.h | 5 +
clang/include/clang/Basic/OpenMPKinds.def | 9 ++
clang/include/clang/Basic/OpenMPKinds.h | 7 ++
clang/include/clang/Sema/SemaOpenMP.h | 6 ++
clang/lib/AST/OpenMPClause.cpp | 8 ++
clang/lib/AST/StmtProfile.cpp | 2 +
clang/lib/Basic/OpenMPKinds.cpp | 15 +++
clang/lib/CodeGen/CGOpenMPRuntime.cpp | 18 ++--
clang/lib/CodeGen/CGOpenMPRuntime.h | 9 +-
clang/lib/CodeGen/CGStmtOpenMP.cpp | 89 +++++++++------
clang/lib/Parse/ParseOpenMP.cpp | 1 +
clang/lib/Sema/SemaOpenMP.cpp | 29 ++++-
clang/lib/Sema/TreeTransform.h | 21 ++++
clang/lib/Serialization/ASTReader.cpp | 9 ++
clang/lib/Serialization/ASTWriter.cpp | 6 ++
clang/test/OpenMP/flush_memscope_codegen.cpp | 101 ++++++++++++++++++
clang/test/OpenMP/memscope_ast_print.cpp | 44 ++++++++
clang/test/OpenMP/memscope_messages.cpp | 26 +++++
clang/tools/libclang/CIndex.cpp | 2 +
llvm/include/llvm/Frontend/OpenMP/OMP.td | 3 +
.../llvm/Frontend/OpenMP/OMPIRBuilder.h | 10 +-
llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp | 14 +--
.../Frontend/OpenMPIRBuilderTest.cpp | 31 ++++++
26 files changed, 499 insertions(+), 56 deletions(-)
create mode 100644 clang/test/OpenMP/flush_memscope_codegen.cpp
create mode 100644 clang/test/OpenMP/memscope_ast_print.cpp
create mode 100644 clang/test/OpenMP/memscope_messages.cpp
diff --git a/clang/docs/OpenMPSupport.md b/clang/docs/OpenMPSupport.md
index 3e4311df9e4a18..218a62652cd145 100644
--- a/clang/docs/OpenMPSupport.md
+++ b/clang/docs/OpenMPSupport.md
@@ -309,7 +309,7 @@ implementation.
| depth clause on loop fuse | {none}`unclaimed` | {none}`unclaimed` | |
| workdistribute construct | | {none}`in progress` | @skc7, @mjklemm |
| task_iteration | {none}`unclaimed` | {none}`unclaimed` | |
-| memscope clause for atomic and flush | {none}`unclaimed` | {none}`unclaimed` | |
+| memscope clause for atomic and flush | {part}`partial` | {none}`unclaimed` | Clang: flush, and the flushes implied by atomic (the atomic operation itself stays system-scope) |
| transparent clause (hull tasks) | {part}`partial` | {none}`unclaimed` | Clang parsing/sema [PR174646][PR174646] |
| rule-based compound directives | {part}`In Progress` | {part}`In Progress` | kparzysz Testing for Fortran missing |
| C23, C++23 | {none}`unclaimed` | | |
diff --git a/clang/docs/ReleaseNotes.md b/clang/docs/ReleaseNotes.md
index c0e602b72bd0d4..a1c2fb8ac0fab4 100644
--- a/clang/docs/ReleaseNotes.md
+++ b/clang/docs/ReleaseNotes.md
@@ -1589,9 +1589,9 @@ The `alpha.cplusplus.UseAfterLifetimeEnd` checker was renamed to `alpha.core.Use
another tile's intra-tile loop.
- The `acquire`, `release` and `acq_rel` clauses of the `flush` directive now
- take effect. On GPUs a flush is emitted as a fence of the requested ordering.
- On the host, a flush other than a seq_cst flush needs an OpenMP runtime from
- this release.
+ take effect, and the OpenMP 6.0 `memscope` clause is supported on `flush` and
+ `atomic`. On the host, a flush other than a seq_cst flush at device scope needs
+ an OpenMP runtime from this release.
- Added parsing and semantic support for `dims` modifier in `num_teams`,
`thread_limit` and `num_threads` clauses for OpenMP 6.1 or later.
- Map-type-modifying modifiers applied to a list item with a user-defined mapper
diff --git a/clang/include/clang/AST/OpenMPClause.h b/clang/include/clang/AST/OpenMPClause.h
index 3fc2e696b1c2d0..19d3379d06c4a4 100644
--- a/clang/include/clang/AST/OpenMPClause.h
+++ b/clang/include/clang/AST/OpenMPClause.h
@@ -2400,6 +2400,88 @@ class OMPSeverityClause final : public OMPClause {
}
};
+/// This represents the 'memscope' clause in the '#pragma omp atomic' and
+/// '#pragma omp flush' directives.
+///
+/// \code
+/// #pragma omp flush release memscope(all)
+/// \endcode
+/// In this example directive '#pragma omp flush' has simple 'memscope' clause
+/// with scope specifier 'all'.
+class OMPMemscopeClause final : public OMPClause {
+ friend class OMPClauseReader;
+
+ /// Location of '('
+ SourceLocation LParenLoc;
+
+ /// A kind of the 'memscope' clause.
+ OpenMPMemscopeClauseKind Kind = OMPC_MEMSCOPE_unknown;
+
+ /// Start location of the kind in source code.
+ SourceLocation KindKwLoc;
+
+ /// Set kind of the clause.
+ ///
+ /// \param K Kind of clause.
+ void setMemscopeKind(OpenMPMemscopeClauseKind K) { Kind = K; }
+
+ /// Set clause kind location.
+ ///
+ /// \param KLoc Kind location.
+ void setMemscopeKindKwLoc(SourceLocation KLoc) { KindKwLoc = KLoc; }
+
+ /// Sets the location of '('.
+ void setLParenLoc(SourceLocation Loc) { LParenLoc = Loc; }
+
+public:
+ /// Build 'memscope' clause with argument \a A ('all', 'cgroup' or
+ /// 'device').
+ ///
+ /// \param A Argument of the clause ('all', 'cgroup' or 'device').
+ /// \param ALoc Starting location of the argument.
+ /// \param StartLoc Starting location of the clause.
+ /// \param LParenLoc Location of '('.
+ /// \param EndLoc Ending location of the clause.
+ OMPMemscopeClause(OpenMPMemscopeClauseKind A, SourceLocation ALoc,
+ SourceLocation StartLoc, SourceLocation LParenLoc,
+ SourceLocation EndLoc)
+ : OMPClause(llvm::omp::OMPC_memscope, StartLoc, EndLoc),
+ LParenLoc(LParenLoc), Kind(A), KindKwLoc(ALoc) {}
+
+ /// Build an empty clause.
+ OMPMemscopeClause()
+ : OMPClause(llvm::omp::OMPC_memscope, SourceLocation(),
+ SourceLocation()) {}
+
+ /// Returns the location of '('.
+ SourceLocation getLParenLoc() const { return LParenLoc; }
+
+ /// Returns kind of the clause.
+ OpenMPMemscopeClauseKind getMemscopeKind() const { return Kind; }
+
+ /// Returns location of clause kind.
+ SourceLocation getMemscopeKindKwLoc() const { return KindKwLoc; }
+
+ child_range children() {
+ return child_range(child_iterator(), child_iterator());
+ }
+
+ const_child_range children() const {
+ return const_child_range(const_child_iterator(), const_child_iterator());
+ }
+
+ child_range used_children() {
+ return child_range(child_iterator(), child_iterator());
+ }
+ const_child_range used_children() const {
+ return const_child_range(const_child_iterator(), const_child_iterator());
+ }
+
+ static bool classof(const OMPClause *T) {
+ return T->getClauseKind() == llvm::omp::OMPC_memscope;
+ }
+};
+
/// This represents the 'message' clause in the '#pragma omp error' and the
/// '#pragma omp parallel' directives.
///
diff --git a/clang/include/clang/AST/RecursiveASTVisitor.h b/clang/include/clang/AST/RecursiveASTVisitor.h
index 56aa9b772b1e75..39aa3dbc5b9816 100644
--- a/clang/include/clang/AST/RecursiveASTVisitor.h
+++ b/clang/include/clang/AST/RecursiveASTVisitor.h
@@ -3677,6 +3677,11 @@ bool RecursiveASTVisitor<Derived>::VisitOMPSeverityClause(OMPSeverityClause *) {
return true;
}
+template <typename Derived>
+bool RecursiveASTVisitor<Derived>::VisitOMPMemscopeClause(OMPMemscopeClause *) {
+ return true;
+}
+
template <typename Derived>
bool RecursiveASTVisitor<Derived>::VisitOMPMessageClause(OMPMessageClause *C) {
TRY_TO(TraverseStmt(C->getMessageString()));
diff --git a/clang/include/clang/Basic/OpenMPKinds.def b/clang/include/clang/Basic/OpenMPKinds.def
index 5b5222a2ab4463..a29bf3cfb474fd 100644
--- a/clang/include/clang/Basic/OpenMPKinds.def
+++ b/clang/include/clang/Basic/OpenMPKinds.def
@@ -53,6 +53,9 @@
#ifndef OPENMP_SEVERITY_KIND
#define OPENMP_SEVERITY_KIND(Name)
#endif
+#ifndef OPENMP_MEMSCOPE_KIND
+#define OPENMP_MEMSCOPE_KIND(Name)
+#endif
#ifndef OPENMP_DEFAULTMAP_MODIFIER
#define OPENMP_DEFAULTMAP_MODIFIER(Name)
#endif
@@ -199,6 +202,11 @@ OPENMP_AT_KIND(execution)
OPENMP_SEVERITY_KIND(fatal)
OPENMP_SEVERITY_KIND(warning)
+// Scope specifiers for 'memscope' clause.
+OPENMP_MEMSCOPE_KIND(all)
+OPENMP_MEMSCOPE_KIND(cgroup)
+OPENMP_MEMSCOPE_KIND(device)
+
// Map types for 'map' clause.
OPENMP_MAP_KIND(alloc)
OPENMP_MAP_KIND(to)
@@ -330,6 +338,7 @@ OPENMP_USE_DEVICE_PTR_FALLBACK_MODIFIER(fb_preserve)
#undef OPENMP_ATOMIC_FAIL_MODIFIER
#undef OPENMP_AT_KIND
#undef OPENMP_SEVERITY_KIND
+#undef OPENMP_MEMSCOPE_KIND
#undef OPENMP_MAP_KIND
#undef OPENMP_MAP_MODIFIER_KIND
#undef OPENMP_MOTION_MODIFIER_KIND
diff --git a/clang/include/clang/Basic/OpenMPKinds.h b/clang/include/clang/Basic/OpenMPKinds.h
index a10d2d4bb34473..1284f0aa659cb5 100644
--- a/clang/include/clang/Basic/OpenMPKinds.h
+++ b/clang/include/clang/Basic/OpenMPKinds.h
@@ -153,6 +153,13 @@ enum OpenMPSeverityClauseKind {
OMPC_SEVERITY_unknown
};
+/// OpenMP scope specifiers for 'memscope' clause.
+enum OpenMPMemscopeClauseKind {
+#define OPENMP_MEMSCOPE_KIND(Name) OMPC_MEMSCOPE_##Name,
+#include "clang/Basic/OpenMPKinds.def"
+ OMPC_MEMSCOPE_unknown
+};
+
/// OpenMP device type for 'device_type' clause.
enum OpenMPDeviceType {
#define OPENMP_DEVICE_TYPE_KIND(Name) \
diff --git a/clang/include/clang/Sema/SemaOpenMP.h b/clang/include/clang/Sema/SemaOpenMP.h
index a8a291d02722ce..4926200e987a55 100644
--- a/clang/include/clang/Sema/SemaOpenMP.h
+++ b/clang/include/clang/Sema/SemaOpenMP.h
@@ -1200,6 +1200,12 @@ class SemaOpenMP : public SemaBase {
SourceLocation StartLoc,
SourceLocation LParenLoc,
SourceLocation EndLoc);
+ /// Called on well-formed 'memscope' clause.
+ OMPClause *ActOnOpenMPMemscopeClause(OpenMPMemscopeClauseKind Kind,
+ SourceLocation KindLoc,
+ SourceLocation StartLoc,
+ SourceLocation LParenLoc,
+ SourceLocation EndLoc);
/// Called on well-formed 'message' clause.
/// passing string for message.
diff --git a/clang/lib/AST/OpenMPClause.cpp b/clang/lib/AST/OpenMPClause.cpp
index f0102720f7d33b..89bfee5ad6c716 100644
--- a/clang/lib/AST/OpenMPClause.cpp
+++ b/clang/lib/AST/OpenMPClause.cpp
@@ -180,6 +180,7 @@ const OMPClauseWithPreInit *OMPClauseWithPreInit::get(const OMPClause *C) {
case OMPC_self_maps:
case OMPC_at:
case OMPC_severity:
+ case OMPC_memscope:
case OMPC_device_type:
case OMPC_match:
case OMPC_nontemporal:
@@ -285,6 +286,7 @@ const OMPClauseWithPostUpdate *OMPClauseWithPostUpdate::get(const OMPClause *C)
case OMPC_self_maps:
case OMPC_at:
case OMPC_severity:
+ case OMPC_memscope:
case OMPC_message:
case OMPC_device_type:
case OMPC_match:
@@ -2173,6 +2175,12 @@ void OMPClausePrinter::VisitOMPSeverityClause(OMPSeverityClause *Node) {
<< ")";
}
+void OMPClausePrinter::VisitOMPMemscopeClause(OMPMemscopeClause *Node) {
+ OS << "memscope("
+ << getOpenMPSimpleClauseTypeName(OMPC_memscope, Node->getMemscopeKind())
+ << ")";
+}
+
void OMPClausePrinter::VisitOMPMessageClause(OMPMessageClause *Node) {
OS << "message(";
if (Expr *E = Node->getMessageString())
diff --git a/clang/lib/AST/StmtProfile.cpp b/clang/lib/AST/StmtProfile.cpp
index 476610b9956c71..20a0508da38f22 100644
--- a/clang/lib/AST/StmtProfile.cpp
+++ b/clang/lib/AST/StmtProfile.cpp
@@ -607,6 +607,8 @@ void OMPClauseProfiler::VisitOMPAtClause(const OMPAtClause *C) {}
void OMPClauseProfiler::VisitOMPSeverityClause(const OMPSeverityClause *C) {}
+void OMPClauseProfiler::VisitOMPMemscopeClause(const OMPMemscopeClause *C) {}
+
void OMPClauseProfiler::VisitOMPMessageClause(const OMPMessageClause *C) {
if (C->getMessageString())
Profiler->VisitStmt(C->getMessageString());
diff --git a/clang/lib/Basic/OpenMPKinds.cpp b/clang/lib/Basic/OpenMPKinds.cpp
index 3d06da48b03dcf..8f19f5598826ca 100644
--- a/clang/lib/Basic/OpenMPKinds.cpp
+++ b/clang/lib/Basic/OpenMPKinds.cpp
@@ -149,6 +149,11 @@ unsigned clang::getOpenMPSimpleClauseType(OpenMPClauseKind Kind, StringRef Str,
#define OPENMP_SEVERITY_KIND(Name) .Case(#Name, OMPC_SEVERITY_##Name)
#include "clang/Basic/OpenMPKinds.def"
.Default(OMPC_SEVERITY_unknown);
+ case OMPC_memscope:
+ return llvm::StringSwitch<OpenMPMemscopeClauseKind>(Str)
+#define OPENMP_MEMSCOPE_KIND(Name) .Case(#Name, OMPC_MEMSCOPE_##Name)
+#include "clang/Basic/OpenMPKinds.def"
+ .Default(OMPC_MEMSCOPE_unknown);
case OMPC_lastprivate:
return llvm::StringSwitch<OpenMPLastprivateModifier>(Str)
#define OPENMP_LASTPRIVATE_KIND(Name) .Case(#Name, OMPC_LASTPRIVATE_##Name)
@@ -493,6 +498,16 @@ const char *clang::getOpenMPSimpleClauseTypeName(OpenMPClauseKind Kind,
#include "clang/Basic/OpenMPKinds.def"
}
llvm_unreachable("Invalid OpenMP 'severity' clause type");
+ case OMPC_memscope:
+ switch (Type) {
+ case OMPC_MEMSCOPE_unknown:
+ return "unknown";
+#define OPENMP_MEMSCOPE_KIND(Name) \
+ case OMPC_MEMSCOPE_##Name: \
+ return #Name;
+#include "clang/Basic/OpenMPKinds.def"
+ }
+ llvm_unreachable("Invalid OpenMP 'memscope' clause type");
case OMPC_lastprivate:
switch (Type) {
case OMPC_LASTPRIVATE_unknown:
diff --git a/clang/lib/CodeGen/CGOpenMPRuntime.cpp b/clang/lib/CodeGen/CGOpenMPRuntime.cpp
index 9e6d3e0c5a5199..324bd0ea4ffa5d 100644
--- a/clang/lib/CodeGen/CGOpenMPRuntime.cpp
+++ b/clang/lib/CodeGen/CGOpenMPRuntime.cpp
@@ -2952,10 +2952,11 @@ void CGOpenMPRuntime::emitProcBindClause(CodeGenFunction &CGF,
}
void CGOpenMPRuntime::emitFlush(CodeGenFunction &CGF, ArrayRef<const Expr *>,
- SourceLocation Loc, llvm::AtomicOrdering AO) {
+ SourceLocation Loc, llvm::AtomicOrdering AO,
+ SyncScope Scope) {
// The OpenMPIRBuilder emits a fence on a GPU, so GPUs always use it.
if (CGF.CGM.getLangOpts().OpenMPIRBuilder || isGPU()) {
- OMPBuilder.createFlush(CGF.Builder, AO);
+ OMPBuilder.createFlush(CGF.Builder, AO, getAtomicScope(Scope));
return;
}
if (!CGF.HaveInsertPoint())
@@ -2966,7 +2967,8 @@ void CGOpenMPRuntime::emitFlush(CodeGenFunction &CGF, ArrayRef<const Expr *>,
// A flush with a list is a strong flush, which is treated as seq_cst.
if (AO == llvm::AtomicOrdering::NotAtomic)
AO = llvm::AtomicOrdering::SequentiallyConsistent;
- if (AO == llvm::AtomicOrdering::SequentiallyConsistent) {
+ if (AO == llvm::AtomicOrdering::SequentiallyConsistent &&
+ Scope == SyncScope::DeviceScope) {
// Build call void __kmpc_flush(ident_t *loc)
CGF.EmitRuntimeCall(OMPBuilder.getOrCreateRuntimeFunction(
CGM.getModule(), OMPRTL___kmpc_flush),
@@ -2974,12 +2976,12 @@ void CGOpenMPRuntime::emitFlush(CodeGenFunction &CGF, ArrayRef<const Expr *>,
return;
}
// Build call void __kmpc_flush_explicit(ident_t *loc, kmp_int32 order,
- // kmp_int32 scope), where order is an __ATOMIC_* value and scope is the
- // __MEMORY_SCOPE_* value for device scope.
+ // kmp_int32 scope), where order is an __ATOMIC_* value and scope is a
+ // __MEMORY_SCOPE_* value.
llvm::Value *Args[] = {
emitUpdateLocation(CGF, Loc),
CGF.Builder.getInt32(static_cast<unsigned>(llvm::toCABI(AO))),
- CGF.Builder.getInt32(static_cast<unsigned>(llvm::AtomicScope::Device))};
+ CGF.Builder.getInt32(static_cast<unsigned>(getAtomicScope(Scope)))};
CGF.EmitRuntimeCall(OMPBuilder.getOrCreateRuntimeFunction(
CGM.getModule(), OMPRTL___kmpc_flush_explicit),
Args);
@@ -13605,8 +13607,8 @@ Address CGOpenMPSIMDRuntime::getAddrOfArtificialThreadPrivate(
void CGOpenMPSIMDRuntime::emitFlush(CodeGenFunction &CGF,
ArrayRef<const Expr *> Vars,
- SourceLocation Loc,
- llvm::AtomicOrdering AO) {
+ SourceLocation Loc, llvm::AtomicOrdering AO,
+ SyncScope Scope) {
llvm_unreachable("Not supported in SIMD-only mode");
}
diff --git a/clang/lib/CodeGen/CGOpenMPRuntime.h b/clang/lib/CodeGen/CGOpenMPRuntime.h
index 2f719aafccc9cc..ae257f351aecc4 100644
--- a/clang/lib/CodeGen/CGOpenMPRuntime.h
+++ b/clang/lib/CodeGen/CGOpenMPRuntime.h
@@ -19,6 +19,7 @@
#include "clang/AST/Type.h"
#include "clang/Basic/OpenMPKinds.h"
#include "clang/Basic/SourceLocation.h"
+#include "clang/Basic/SyncScope.h"
#include "llvm/ADT/DenseMap.h"
#include "llvm/ADT/PointerIntPair.h"
#include "llvm/ADT/SmallPtrSet.h"
@@ -1143,8 +1144,11 @@ class CGOpenMPRuntime {
/// \param Vars List of variables to flush.
/// \param AO Memory ordering of the flush, or NotAtomic for a flush with a
/// list (a strong flush).
+ /// \param Scope Memory scope of the flush, as given by the 'memscope'
+ /// clause.
virtual void emitFlush(CodeGenFunction &CGF, ArrayRef<const Expr *> Vars,
- SourceLocation Loc, llvm::AtomicOrdering AO);
+ SourceLocation Loc, llvm::AtomicOrdering AO,
+ SyncScope Scope);
/// Emit task region for the task directive. The task region is
/// emitted in several steps:
@@ -2028,7 +2032,8 @@ class CGOpenMPSIMDRuntime final : public CGOpenMPRuntime {
/// Emit flush of the variables specified in 'omp flush' directive.
/// \param Vars List of variables to flush.
void emitFlush(CodeGenFunction &CGF, ArrayRef<const Expr *> Vars,
- SourceLocation Loc, llvm::AtomicOrdering AO) override;
+ SourceLocation Loc, llvm::AtomicOrdering AO,
+ SyncScope Scope) override;
/// Emit task region for the task directive. The task region is
/// emitted in several steps:
diff --git a/clang/lib/CodeGen/CGStmtOpenMP.cpp b/clang/lib/CodeGen/CGStmtOpenMP.cpp
index 6959fe4ada619a..ee660c6cf336a4 100644
--- a/clang/lib/CodeGen/CGStmtOpenMP.cpp
+++ b/clang/lib/CodeGen/CGStmtOpenMP.cpp
@@ -6261,6 +6261,26 @@ void CodeGenFunction::EmitOMPTaskgroupDirective(
CGM.getOpenMPRuntime().emitTaskgroupRegion(*this, CodeGen, S.getBeginLoc());
}
+/// Returns the memory scope given by the 'memscope' clause of \p S, or the
+/// device scope that applies when there is no such clause.
+static SyncScope getOMPMemscope(const OMPExecutableDirective &S) {
+ if (const auto *C = S.getSingleClause<OMPMemscopeClause>()) {
+ switch (C->getMemscopeKind()) {
+ case OMPC_MEMSCOPE_all:
+ return SyncScope::SystemScope;
+ case OMPC_MEMSCOPE_cgroup:
+ // A contention group is the threads of one team, and clang maps each
+ // team to one workgroup on a GPU.
+ return SyncScope::WorkgroupScope;
+ case OMPC_MEMSCOPE_device:
+ return SyncScope::DeviceScope;
+ case OMPC_MEMSCOPE_unknown:
+ llvm_unreachable("Unexpected memscope clause kind.");
+ }
+ }
+ return SyncScope::DeviceScope;
+}
+
void CodeGenFunction::EmitOMPFlushDirective(const OMPFlushDirective &S) {
// A flush with a list is a strong flush. Without a list the flush behaves as
// if the memory-order clause is seq_cst unless another one is given.
@@ -6281,7 +6301,7 @@ void CodeGenFunction::EmitOMPFlushDirective(const OMPFlushDirective &S) {
FlushClause->varlist_end());
return {};
}(),
- S.getBeginLoc(), AO);
+ S.getBeginLoc(), AO, getOMPMemscope(S));
}
void CodeGenFunction::EmitOMPDepobjDirective(const OMPDepobjDirective &S) {
@@ -6971,7 +6991,7 @@ void CodeGenFunction::emitOMPSimpleStore(LValue LVal, RValue RVal,
}
static void emitOMPAtomicReadExpr(CodeGenFunction &CGF, llvm::AtomicOrdering AO,
- const Expr *X, const Expr *V,
+ SyncScope Scope, const Expr *X, const Expr *V,
SourceLocation Loc) {
// v = x;
assert(V->isLValue() && "V of 'omp atomic read' is not lvalue");
@@ -6991,7 +7011,7 @@ static void emitOMPAtomicReadExpr(CodeGenFunction &CGF, llvm::AtomicOrdering AO,
case llvm::AtomicOrdering::AcquireRelease:
case llvm::AtomicOrdering::SequentiallyConsistent:
CGF.CGM.getOpenMPRuntime().emitFlush(
- CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
+ CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent, Scope);
break;
case llvm::AtomicOrdering::Monotonic:
case llvm::AtomicOrdering::Release:
@@ -7005,8 +7025,9 @@ static void emitOMPAtomicReadExpr(CodeGenFunction &CGF, llvm::AtomicOrdering AO,
}
static void emitOMPAtomicWriteExpr(CodeGenFunction &CGF,
- llvm::AtomicOrdering AO, const Expr *X,
- const Expr *E, SourceLocation Loc) {
+ llvm::AtomicOrdering AO, SyncScope Scope,
+ const Expr *X, const Expr *E,
+ SourceLocation Loc) {
// x = expr;
assert(X->isLValue() && "X of 'omp atomic write' is not lvalue");
emitSimpleAtomicStore(CGF, AO, CGF.EmitLValue(X), CGF.EmitAnyExpr(E));
@@ -7020,7 +7041,7 @@ static void emitOMPAtomicWriteExpr(CodeGenFunction &CGF,
case llvm::AtomicOrdering::AcquireRelease:
case llvm::AtomicOrdering::SequentiallyConsistent:
CGF.CGM.getOpenMPRuntime().emitFlush(
- CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
+ CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent, Scope);
break;
case llvm::AtomicOrdering::Acquire:
case llvm::AtomicOrdering::Monotonic:
@@ -7174,9 +7195,10 @@ std::pair<bool, RValue> CodeGenFunction::EmitOMPAtomicSimpleUpdateExpr(
}
static void emitOMPAtomicUpdateExpr(CodeGenFunction &CGF,
- llvm::AtomicOrdering AO, const Expr *X,
- const Expr *E, const Expr *UE,
- bool IsXLHSInRHSPart, SourceLocation Loc) {
+ llvm::AtomicOrdering AO, SyncScope Scope,
+ const Expr *X, const Expr *E,
+ const Expr *UE, bool IsXLHSInRHSPart,
+ SourceLocation Loc) {
assert(isa<BinaryOperator>(UE->IgnoreImpCasts()) &&
"Update expr in 'atomic update' must be a binary operator.");
const auto *BOUE = cast<BinaryOperator>(UE->IgnoreImpCasts());
@@ -7210,7 +7232,7 @@ static void emitOMPAtomicUpdateExpr(CodeGenFunction &CGF,
case llvm::AtomicOrdering::AcquireRelease:
case llvm::AtomicOrdering::SequentiallyConsistent:
CGF.CGM.getOpenMPRuntime().emitFlush(
- CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
+ CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent, Scope);
break;
case llvm::AtomicOrdering::Acquire:
case llvm::AtomicOrdering::Monotonic:
@@ -7239,7 +7261,7 @@ static RValue convertToType(CodeGenFunction &CGF, RValue Value,
}
static void emitOMPAtomicCaptureExpr(CodeGenFunction &CGF,
- llvm::AtomicOrdering AO,
+ llvm::AtomicOrdering AO, SyncScope Scope,
bool IsPostfixUpdate, const Expr *V,
const Expr *X, const Expr *E,
const Expr *UE, bool IsXLHSInRHSPart,
@@ -7328,7 +7350,7 @@ static void emitOMPAtomicCaptureExpr(CodeGenFunction &CGF,
case llvm::AtomicOrdering::AcquireRelease:
case llvm::AtomicOrdering::SequentiallyConsistent:
CGF.CGM.getOpenMPRuntime().emitFlush(
- CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent);
+ CGF, {}, Loc, llvm::AtomicOrdering::SequentiallyConsistent, Scope);
break;
case llvm::AtomicOrdering::Monotonic:
break;
@@ -7341,9 +7363,9 @@ static void emitOMPAtomicCaptureExpr(CodeGenFunction &CGF,
static void emitOMPAtomicCompareExpr(
CodeGenFunction &CGF, llvm::AtomicOrdering AO, llvm::AtomicOrdering FailAO,
- const Expr *X, const Expr *V, const Expr *R, const Expr *E, const Expr *D,
- const Expr *CE, bool IsXBinopExpr, bool IsPostfixUpdate, bool IsFailOnly,
- SourceLocation Loc) {
+ SyncScope Scope, const Expr *X, const Expr *V, const Expr *R, const Expr *E,
+ const Expr *D, const Expr *CE, bool IsXBinopExpr, bool IsPostfixUpdate,
+ bool IsFailOnly, SourceLocation Loc) {
llvm::OpenMPIRBuilder &OMPBuilder =
CGF.CGM.getOpenMPRuntime().getOMPBuilder();
@@ -7413,37 +7435,39 @@ static void emitOMPAtomicCompareExpr(
// "#pragma omp atomic compare" construct.
CGF.Builder.restoreIP(OMPBuilder.createAtomicCompare(
CGF.Builder, XOpVal, VOpVal, ROpVal, EVal, DVal, AO, Op, IsXBinopExpr,
- IsPostfixUpdate, IsFailOnly));
+ IsPostfixUpdate, IsFailOnly, /*IsWeak=*/false, getAtomicScope(Scope)));
} else
CGF.Builder.restoreIP(OMPBuilder.createAtomicCompare(
CGF.Builder, XOpVal, VOpVal, ROpVal, EVal, DVal, AO, Op, IsXBinopExpr,
- IsPostfixUpdate, IsFailOnly, FailAO));
+ IsPostfixUpdate, IsFailOnly, FailAO, /*IsWeak=*/false,
+ getAtomicScope(Scope)));
}
static void emitOMPAtomicExpr(CodeGenFunction &CGF, OpenMPClauseKind Kind,
llvm::AtomicOrdering AO,
- llvm::AtomicOrdering FailAO, bool IsPostfixUpdate,
- const Expr *X, const Expr *V, const Expr *R,
- const Expr *E, const Expr *UE, const Expr *D,
- const Expr *CE, bool IsXLHSInRHSPart,
- bool IsFailOnly, SourceLocation Loc) {
+ llvm::AtomicOrdering FailAO, SyncScope Scope,
+ bool IsPostfixUpdate, const Expr *X,
+ const Expr *V, const Expr *R, const Expr *E,
+ const Expr *UE, const Expr *D, const Expr *CE,
+ bool IsXLHSInRHSPart, bool IsFailOnly,
+ SourceLocation Loc) {
switch (Kind) {
case OMPC_read:
- emitOMPAtomicReadExpr(CGF, AO, X, V, Loc);
+ emitOMPAtomicReadExpr(CGF, AO, Scope, X, V, Loc);
break;
case OMPC_write:
- emitOMPAtomicWriteExpr(CGF, AO, X, E, Loc);
+ emitOMPAtomicWriteExpr(CGF, AO, Scope, X, E, Loc);
break;
case OMPC_unknown:
case OMPC_update:
- emitOMPAtomicUpdateExpr(CGF, AO, X, E, UE, IsXLHSInRHSPart, Loc);
+ emitOMPAtomicUpdateExpr(CGF, AO, Scope, X, E, UE, IsXLHSInRHSPart, Loc);
break;
case OMPC_capture:
- emitOMPAtomicCaptureExpr(CGF, AO, IsPostfixUpdate, V, X, E, UE,
+ emitOMPAtomicCaptureExpr(CGF, AO, Scope, IsPostfixUpdate, V, X, E, UE,
IsXLHSInRHSPart, Loc);
break;
case OMPC_compare: {
- emitOMPAtomicCompareExpr(CGF, AO, FailAO, X, V, R, E, D, CE,
+ emitOMPAtomicCompareExpr(CGF, AO, FailAO, Scope, X, V, R, E, D, CE,
IsXLHSInRHSPart, IsPostfixUpdate, IsFailOnly, Loc);
break;
}
@@ -7483,7 +7507,8 @@ void CodeGenFunction::EmitOMPAtomicDirective(const OMPAtomicDirective &S) {
if (K == OMPC_weak)
return;
if (K == OMPC_seq_cst || K == OMPC_acq_rel || K == OMPC_acquire ||
- K == OMPC_release || K == OMPC_relaxed || K == OMPC_hint)
+ K == OMPC_release || K == OMPC_relaxed || K == OMPC_hint ||
+ K == OMPC_memscope)
continue;
Kind = K;
KindsEncountered.insert(K);
@@ -7543,10 +7568,10 @@ void CodeGenFunction::EmitOMPAtomicDirective(const OMPAtomicDirective &S) {
LexicalScope Scope(*this, S.getSourceRange());
EmitStopPoint(S.getAssociatedStmt());
- emitOMPAtomicExpr(*this, Kind, AO, FailAO, S.isPostfixUpdate(), S.getX(),
- S.getV(), S.getR(), S.getExpr(), S.getUpdateExpr(),
- S.getD(), S.getCondExpr(), S.isXLHSInRHSPart(),
- S.isFailOnly(), S.getBeginLoc());
+ emitOMPAtomicExpr(*this, Kind, AO, FailAO, getOMPMemscope(S),
+ S.isPostfixUpdate(), S.getX(), S.getV(), S.getR(),
+ S.getExpr(), S.getUpdateExpr(), S.getD(), S.getCondExpr(),
+ S.isXLHSInRHSPart(), S.isFailOnly(), S.getBeginLoc());
}
static void emitCommonOMPTargetDirective(CodeGenFunction &CGF,
diff --git a/clang/lib/Parse/ParseOpenMP.cpp b/clang/lib/Parse/ParseOpenMP.cpp
index f6fab0b00b0b59..bf799cd6d48343 100644
--- a/clang/lib/Parse/ParseOpenMP.cpp
+++ b/clang/lib/Parse/ParseOpenMP.cpp
@@ -3438,6 +3438,7 @@ OMPClause *Parser::ParseOpenMPClause(OpenMPDirectiveKind DKind,
case OMPC_atomic_default_mem_order:
case OMPC_at:
case OMPC_severity:
+ case OMPC_memscope:
case OMPC_bind:
// OpenMP [2.14.3.1, Restrictions]
// Only a single default clause may be specified on a parallel, task or
diff --git a/clang/lib/Sema/SemaOpenMP.cpp b/clang/lib/Sema/SemaOpenMP.cpp
index 505ebc0ab69331..8cf290fc27a7b8 100644
--- a/clang/lib/Sema/SemaOpenMP.cpp
+++ b/clang/lib/Sema/SemaOpenMP.cpp
@@ -7361,6 +7361,7 @@ StmtResult SemaOpenMP::ActOnOpenMPExecutableDirective(
case OMPC_bind:
case OMPC_filter:
case OMPC_severity:
+ case OMPC_memscope:
case OMPC_message:
continue;
case OMPC_allocator:
@@ -12534,7 +12535,7 @@ StmtResult SemaOpenMP::ActOnOpenMPFlushDirective(ArrayRef<OMPClause *> Clauses,
for (OMPClause *C : Clauses) {
if (C->getClauseKind() == OMPC_flush)
FC = cast<OMPFlushClause>(C);
- else
+ else if (C->getClauseKind() != OMPC_memscope)
OrderClause = C;
}
llvm::omp::Version OMPVersion = getLangOpts().getOpenMPVersion();
@@ -13910,6 +13911,7 @@ StmtResult SemaOpenMP::ActOnOpenMPAtomicDirective(ArrayRef<OMPClause *> Clauses,
}
// The following clauses are allowed, but we don't need to do anything here.
case OMPC_hint:
+ case OMPC_memscope:
break;
default:
llvm_unreachable("unknown clause is encountered");
@@ -18769,6 +18771,7 @@ OMPClause *SemaOpenMP::ActOnOpenMPSingleExprClause(OpenMPClauseKind Kind,
case OMPC_order:
case OMPC_at:
case OMPC_severity:
+ case OMPC_memscope:
case OMPC_destroy:
case OMPC_inclusive:
case OMPC_exclusive:
@@ -19445,6 +19448,11 @@ OMPClause *SemaOpenMP::ActOnOpenMPSimpleClause(
static_cast<OpenMPSeverityClauseKind>(Argument), ArgumentLoc, StartLoc,
LParenLoc, EndLoc);
break;
+ case OMPC_memscope:
+ Res = ActOnOpenMPMemscopeClause(
+ static_cast<OpenMPMemscopeClauseKind>(Argument), ArgumentLoc, StartLoc,
+ LParenLoc, EndLoc);
+ break;
case OMPC_threadset:
Res = ActOnOpenMPThreadsetClause(static_cast<OpenMPThreadsetKind>(Argument),
ArgumentLoc, StartLoc, LParenLoc, EndLoc);
@@ -19818,6 +19826,22 @@ OMPClause *SemaOpenMP::ActOnOpenMPSeverityClause(OpenMPSeverityClauseKind Kind,
OMPSeverityClause(Kind, KindKwLoc, StartLoc, LParenLoc, EndLoc);
}
+OMPClause *SemaOpenMP::ActOnOpenMPMemscopeClause(OpenMPMemscopeClauseKind Kind,
+ SourceLocation KindKwLoc,
+ SourceLocation StartLoc,
+ SourceLocation LParenLoc,
+ SourceLocation EndLoc) {
+ if (Kind == OMPC_MEMSCOPE_unknown) {
+ Diag(KindKwLoc, diag::err_omp_unexpected_clause_value)
+ << getListOfPossibleValues(OMPC_memscope, /*First=*/0,
+ /*Last=*/OMPC_MEMSCOPE_unknown)
+ << getOpenMPClauseNameForDiag(OMPC_memscope);
+ return nullptr;
+ }
+ return new (getASTContext())
+ OMPMemscopeClause(Kind, KindKwLoc, StartLoc, LParenLoc, EndLoc);
+}
+
OMPClause *SemaOpenMP::ActOnOpenMPMessageClause(Expr *ME,
SourceLocation StartLoc,
SourceLocation LParenLoc,
@@ -20271,6 +20295,7 @@ OMPClause *SemaOpenMP::ActOnOpenMPSingleExprWithArgClause(
case OMPC_nontemporal:
case OMPC_at:
case OMPC_severity:
+ case OMPC_memscope:
case OMPC_message:
case OMPC_destroy:
case OMPC_novariants:
@@ -20546,6 +20571,7 @@ OMPClause *SemaOpenMP::ActOnOpenMPClause(OpenMPClauseKind Kind,
case OMPC_order:
case OMPC_at:
case OMPC_severity:
+ case OMPC_memscope:
case OMPC_message:
case OMPC_novariants:
case OMPC_nocontext:
@@ -21218,6 +21244,7 @@ OMPClause *SemaOpenMP::ActOnOpenMPVarListClause(OpenMPClauseKind Kind,
case OMPC_order:
case OMPC_at:
case OMPC_severity:
+ case OMPC_memscope:
case OMPC_message:
case OMPC_destroy:
case OMPC_novariants:
diff --git a/clang/lib/Sema/TreeTransform.h b/clang/lib/Sema/TreeTransform.h
index 2912ed99df4617..76f942045fffa9 100644
--- a/clang/lib/Sema/TreeTransform.h
+++ b/clang/lib/Sema/TreeTransform.h
@@ -2603,6 +2603,19 @@ class TreeTransform {
LParenLoc, EndLoc);
}
+ /// Build a new OpenMP 'memscope' clause.
+ ///
+ /// By default, performs semantic analysis to build the new OpenMP clause.
+ /// Subclasses may override this routine to provide different behavior.
+ OMPClause *RebuildOMPMemscopeClause(OpenMPMemscopeClauseKind Kind,
+ SourceLocation KwLoc,
+ SourceLocation StartLoc,
+ SourceLocation LParenLoc,
+ SourceLocation EndLoc) {
+ return getSema().OpenMP().ActOnOpenMPMemscopeClause(Kind, KwLoc, StartLoc,
+ LParenLoc, EndLoc);
+ }
+
/// Build a new OpenMP 'message' clause.
///
/// By default, performs semantic analysis to build the new OpenMP clause.
@@ -11635,6 +11648,14 @@ TreeTransform<Derived>::TransformOMPSeverityClause(OMPSeverityClause *C) {
C->getLParenLoc(), C->getEndLoc());
}
+template <typename Derived>
+OMPClause *
+TreeTransform<Derived>::TransformOMPMemscopeClause(OMPMemscopeClause *C) {
+ return getDerived().RebuildOMPMemscopeClause(
+ C->getMemscopeKind(), C->getMemscopeKindKwLoc(), C->getBeginLoc(),
+ C->getLParenLoc(), C->getEndLoc());
+}
+
template <typename Derived>
OMPClause *
TreeTransform<Derived>::TransformOMPMessageClause(OMPMessageClause *C) {
diff --git a/clang/lib/Serialization/ASTReader.cpp b/clang/lib/Serialization/ASTReader.cpp
index cd7d9f24924f82..786e05614d91e9 100644
--- a/clang/lib/Serialization/ASTReader.cpp
+++ b/clang/lib/Serialization/ASTReader.cpp
@@ -11825,6 +11825,9 @@ OMPClause *OMPClauseReader::readClause() {
case llvm::omp::OMPC_severity:
C = new (Context) OMPSeverityClause();
break;
+ case llvm::omp::OMPC_memscope:
+ C = new (Context) OMPMemscopeClause();
+ break;
case llvm::omp::OMPC_message:
C = new (Context) OMPMessageClause();
break;
@@ -12392,6 +12395,12 @@ void OMPClauseReader::VisitOMPSeverityClause(OMPSeverityClause *C) {
C->setSeverityKindKwLoc(Record.readSourceLocation());
}
+void OMPClauseReader::VisitOMPMemscopeClause(OMPMemscopeClause *C) {
+ C->setMemscopeKind(static_cast<OpenMPMemscopeClauseKind>(Record.readInt()));
+ C->setLParenLoc(Record.readSourceLocation());
+ C->setMemscopeKindKwLoc(Record.readSourceLocation());
+}
+
void OMPClauseReader::VisitOMPMessageClause(OMPMessageClause *C) {
VisitOMPClauseWithPreInit(C);
C->setMessageString(Record.readSubExpr());
diff --git a/clang/lib/Serialization/ASTWriter.cpp b/clang/lib/Serialization/ASTWriter.cpp
index de1c2449138e58..77341537d71b11 100644
--- a/clang/lib/Serialization/ASTWriter.cpp
+++ b/clang/lib/Serialization/ASTWriter.cpp
@@ -8954,6 +8954,12 @@ void OMPClauseWriter::VisitOMPSeverityClause(OMPSeverityClause *C) {
Record.AddSourceLocation(C->getSeverityKindKwLoc());
}
+void OMPClauseWriter::VisitOMPMemscopeClause(OMPMemscopeClause *C) {
+ Record.push_back(C->getMemscopeKind());
+ Record.AddSourceLocation(C->getLParenLoc());
+ Record.AddSourceLocation(C->getMemscopeKindKwLoc());
+}
+
void OMPClauseWriter::VisitOMPMessageClause(OMPMessageClause *C) {
VisitOMPClauseWithPreInit(C);
Record.AddStmt(C->getMessageString());
diff --git a/clang/test/OpenMP/flush_memscope_codegen.cpp b/clang/test/OpenMP/flush_memscope_codegen.cpp
new file mode 100644
index 00000000000000..8914557ce1ab8f
--- /dev/null
+++ b/clang/test/OpenMP/flush_memscope_codegen.cpp
@@ -0,0 +1,101 @@
+// Check that the OpenMP 6.0 memscope clause sets the scope of a flush and of
+// the flushes implied by 'atomic': a __MEMORY_SCOPE_* value passed to
+// __kmpc_flush_explicit on the host, and the sync scope of the fence on GPUs.
+
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=60 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -o - | FileCheck %s --check-prefix=HOST
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=60 -fopenmp-enable-irbuilder -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -o - | FileCheck %s --check-prefix=HOST
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=60 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm-bc %s -o %t-host.bc
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=60 -x c++ -triple amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host.bc -o - | FileCheck %s --check-prefixes=DEV,AMDGPU
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=60 -fopenmp-enable-irbuilder -x c++ -triple amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host.bc -o - | FileCheck %s --check-prefixes=DEV,AMDGPU
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=60 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=nvptx64-nvidia-cuda -emit-llvm-bc %s -o %t-host-nv.bc
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=60 -x c++ -triple nvptx64-nvidia-cuda -fopenmp-targets=nvptx64-nvidia-cuda -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host-nv.bc -o - | FileCheck %s --check-prefixes=DEV,NVPTX
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=60 -x c++ -triple x86_64-unknown-linux-gnu -fopenmp-targets=spirv64-intel -emit-llvm-bc %s -o %t-host-spirv.bc
+// RUN: %clang_cc1 -fopenmp -fopenmp-version=60 -x c++ -triple spirv64-intel -fopenmp-targets=spirv64-intel -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host-spirv.bc -o - | FileCheck %s --check-prefixes=DEV,SPIRV
+// expected-no-diagnostics
+
+int x, v, e, d;
+
+void foo() {
+#pragma omp target
+ {
+#pragma omp flush memscope(device)
+#pragma omp flush memscope(all)
+#pragma omp flush(x) memscope(all)
+#pragma omp flush release memscope(all)
+#pragma omp flush acquire memscope(cgroup)
+#pragma omp flush acq_rel memscope(device)
+#pragma omp atomic read acquire memscope(cgroup)
+ v = x;
+#pragma omp atomic write release memscope(all)
+ x = v;
+#pragma omp atomic update seq_cst memscope(cgroup)
+ x += v;
+#pragma omp atomic compare seq_cst memscope(all)
+ if (x == e)
+ x = d;
+ }
+}
+
+// memscope(device) is the default, so a seq_cst flush with it still calls
+// __kmpc_flush. Scopes: 0 = system (all), 1 = device, 2 = workgroup (cgroup).
+// HOST-LABEL: define {{.*}}@{{.*}}foo{{.*}}(
+// HOST: call void @__kmpc_flush(ptr
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 5, i32 0)
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 5, i32 0)
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 3, i32 0)
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 2, i32 2)
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 4, i32 1)
+// HOST: load atomic i32, ptr {{.*}} acquire
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 5, i32 2)
+// HOST: store atomic i32 {{.*}} release
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 5, i32 0)
+// HOST: atomicrmw add ptr {{.*}} seq_cst
+// HOST-NEXT: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 5, i32 2)
+// HOST: cmpxchg ptr {{.*}} seq_cst seq_cst
+// HOST: call void @__kmpc_flush_explicit(ptr {{.*}}, i32 5, i32 0)
+
+// DEV-LABEL: define {{.*}}@__omp_offloading_{{.*}}foo{{.*}}(
+// DEV-NOT: call {{.*}}@__kmpc_flush
+// AMDGPU: fence syncscope("agent") seq_cst
+// AMDGPU-NEXT: fence seq_cst
+// AMDGPU-NEXT: fence seq_cst
+// AMDGPU-NEXT: fence release
+// AMDGPU-NEXT: fence syncscope("workgroup") acquire
+// AMDGPU-NEXT: fence syncscope("agent") acq_rel
+// AMDGPU: load atomic i32, ptr {{.*}} acquire
+// AMDGPU-NEXT: fence syncscope("workgroup") seq_cst
+// AMDGPU: store atomic i32 {{.*}} release
+// AMDGPU-NEXT: fence seq_cst
+// AMDGPU: atomicrmw add ptr {{.*}} seq_cst
+// AMDGPU-NEXT: fence syncscope("workgroup") seq_cst
+// AMDGPU: cmpxchg ptr {{.*}} seq_cst seq_cst
+// AMDGPU: fence seq_cst
+// NVPTX: fence syncscope("device") seq_cst
+// NVPTX-NEXT: fence seq_cst
+// NVPTX-NEXT: fence seq_cst
+// NVPTX-NEXT: fence release
+// NVPTX-NEXT: fence syncscope("block") acquire
+// NVPTX-NEXT: fence syncscope("device") acq_rel
+// NVPTX: load atomic i32, ptr {{.*}} acquire
+// NVPTX-NEXT: fence syncscope("block") seq_cst
+// NVPTX: store atomic i32 {{.*}} release
+// NVPTX-NEXT: fence seq_cst
+// NVPTX: atomicrmw add ptr {{.*}} seq_cst
+// NVPTX-NEXT: fence syncscope("block") seq_cst
+// NVPTX: cmpxchg ptr {{.*}} seq_cst seq_cst
+// NVPTX: fence seq_cst
+// SPIRV: fence syncscope("device") seq_cst
+// SPIRV-NEXT: fence seq_cst
+// SPIRV-NEXT: fence seq_cst
+// SPIRV-NEXT: fence release
+// SPIRV-NEXT: fence syncscope("workgroup") acquire
+// SPIRV-NEXT: fence syncscope("device") acq_rel
+// SPIRV: load atomic i32, ptr {{.*}} acquire
+// SPIRV-NEXT: fence syncscope("workgroup") seq_cst
+// SPIRV: store atomic i32 {{.*}} release
+// SPIRV-NEXT: fence seq_cst
+// SPIRV: atomicrmw add ptr {{.*}} seq_cst
+// SPIRV-NEXT: fence syncscope("workgroup") seq_cst
+// SPIRV: cmpxchg ptr {{.*}} seq_cst seq_cst
+// SPIRV: fence seq_cst
+// DEV-NOT: call {{.*}}@__kmpc_flush
diff --git a/clang/test/OpenMP/memscope_ast_print.cpp b/clang/test/OpenMP/memscope_ast_print.cpp
new file mode 100644
index 00000000000000..c23a45bb7a3d0d
--- /dev/null
+++ b/clang/test/OpenMP/memscope_ast_print.cpp
@@ -0,0 +1,44 @@
+// RUN: %clang_cc1 -verify -fopenmp -fopenmp-version=60 -ast-print %s | FileCheck %s
+// RUN: %clang_cc1 -fopenmp -x c++ -std=c++11 -fopenmp-version=60 -emit-pch -o %t %s
+// RUN: %clang_cc1 -fopenmp -std=c++11 -fopenmp-version=60 -include-pch %t -verify %s -ast-print | FileCheck %s
+// RUN: %clang_cc1 -fopenmp -std=c++11 -fopenmp-version=60 -include-pch %t -ast-dump-all %s | FileCheck %s --check-prefix=DUMP
+// expected-no-diagnostics
+
+#ifndef HEADER
+#define HEADER
+
+template <class T> T tmain(T x) {
+ T v;
+#pragma omp flush memscope(all)
+#pragma omp flush release memscope(cgroup)
+#pragma omp atomic read acquire memscope(device)
+ v = x;
+ return v;
+}
+
+// CHECK-LABEL: template <class T> T tmain(T x) {
+// CHECK: #pragma omp flush memscope(all)
+// CHECK-NEXT: #pragma omp flush release memscope(cgroup)
+// CHECK-NEXT: #pragma omp atomic read acquire memscope(device)
+// CHECK-NEXT: v = x;
+
+// CHECK-LABEL: template<> int tmain<int>(int x) {
+// CHECK: #pragma omp flush memscope(all)
+// CHECK-NEXT: #pragma omp flush release memscope(cgroup)
+// CHECK-NEXT: #pragma omp atomic read acquire memscope(device)
+// CHECK-NEXT: v = x;
+
+// DUMP-LABEL: FunctionDecl {{.*}} tmain 'int (int)'
+// DUMP: OMPFlushDirective
+// DUMP-NEXT: OMPMemscopeClause
+// DUMP: OMPAtomicDirective
+// DUMP-NEXT: OMPReadClause
+// DUMP-NEXT: OMPAcquireClause
+// DUMP-NEXT: OMPMemscopeClause
+
+int main() {
+ int x = 0;
+ return tmain(x);
+}
+
+#endif
diff --git a/clang/test/OpenMP/memscope_messages.cpp b/clang/test/OpenMP/memscope_messages.cpp
new file mode 100644
index 00000000000000..10aeec3d22e616
--- /dev/null
+++ b/clang/test/OpenMP/memscope_messages.cpp
@@ -0,0 +1,26 @@
+// RUN: %clang_cc1 -verify=expected,omp51 -fopenmp -fopenmp-version=51 -ferror-limit 100 %s
+// RUN: %clang_cc1 -verify=expected,omp60 -fopenmp -fopenmp-version=60 -ferror-limit 100 %s
+
+// RUN: %clang_cc1 -verify=expected,omp51 -fopenmp-simd -fopenmp-version=51 -ferror-limit 100 %s
+// RUN: %clang_cc1 -verify=expected,omp60 -fopenmp-simd -fopenmp-version=60 -ferror-limit 100 %s
+
+int x, v;
+
+void foo() {
+#pragma omp flush memscope(device) // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp flush'}}
+#pragma omp flush release memscope(all) // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp flush'}}
+#pragma omp flush memscope(cgroup) acquire // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp flush'}}
+#pragma omp flush memscope(system) // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp flush'}} omp60-error {{expected 'all', 'cgroup' or 'device' in OpenMP clause 'memscope'}}
+#pragma omp flush memscope( // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp flush'}} omp60-error {{expected 'all', 'cgroup' or 'device' in OpenMP clause 'memscope'}} expected-error {{expected ')'}} expected-note {{to match this '('}}
+#pragma omp flush memscope // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp flush'}} expected-error {{expected '(' after 'memscope'}}
+#pragma omp flush memscope(all) memscope(device) // omp51-error 2 {{unexpected OpenMP clause 'memscope' in directive '#pragma omp flush'}} expected-error {{directive '#pragma omp flush' cannot contain more than one 'memscope' clause}}
+#pragma omp flush(x) memscope(all) // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp flush'}}
+#pragma omp flush(x) release memscope(all) // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp flush'}} expected-error {{'flush' directive with memory order clause 'release' cannot have the list}} expected-note {{memory order clause 'release' is specified here}}
+#pragma omp atomic read memscope(device) // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp atomic'}}
+ v = x;
+#pragma omp atomic seq_cst memscope(all) // omp51-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp atomic'}}
+ x += v;
+#pragma omp atomic memscope(cgroup) memscope(all) // omp51-error 2 {{unexpected OpenMP clause 'memscope' in directive '#pragma omp atomic'}} expected-error {{directive '#pragma omp atomic' cannot contain more than one 'memscope' clause}}
+ x++;
+#pragma omp barrier memscope(all) // expected-error {{unexpected OpenMP clause 'memscope' in directive '#pragma omp barrier'}}
+}
diff --git a/clang/tools/libclang/CIndex.cpp b/clang/tools/libclang/CIndex.cpp
index a80a61f6492767..b1c6f2650a5490 100644
--- a/clang/tools/libclang/CIndex.cpp
+++ b/clang/tools/libclang/CIndex.cpp
@@ -2550,6 +2550,8 @@ void OMPClauseEnqueue::VisitOMPAtClause(const OMPAtClause *) {}
void OMPClauseEnqueue::VisitOMPSeverityClause(const OMPSeverityClause *) {}
+void OMPClauseEnqueue::VisitOMPMemscopeClause(const OMPMemscopeClause *) {}
+
void OMPClauseEnqueue::VisitOMPMessageClause(const OMPMessageClause *) {}
void OMPClauseEnqueue::VisitOMPDeviceClause(const OMPDeviceClause *C) {
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMP.td b/llvm/include/llvm/Frontend/OpenMP/OMP.td
index 660ab03f9516cd..af29411f4997c5 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMP.td
+++ b/llvm/include/llvm/Frontend/OpenMP/OMP.td
@@ -339,6 +339,7 @@ def OMPC_Match : Clause<[Spelling<"match">]> {
let flangClass = "OmpMatchClause";
}
def OMPC_Memscope : Clause<[Spelling<"memscope">]> {
+ let clangClass = "OMPMemscopeClause";
}
def OMP_MEMORY_ORDER_SeqCst : EnumVal<"seq_cst", 1, 1> {}
def OMP_MEMORY_ORDER_AcqRel : EnumVal<"acq_rel", 2, 1> {}
@@ -716,6 +717,7 @@ def OMP_Atomic : Directive<[Spelling<"atomic">]> {
VersionedClause<OMPC_Compare, 51>,
VersionedClause<OMPC_Fail, 51>,
VersionedClause<OMPC_Hint, 50>,
+ VersionedClause<OMPC_Memscope, 60>,
VersionedClause<OMPC_Read>,
VersionedClause<OMPC_Relaxed, 50>,
VersionedClause<OMPC_Release, 50>,
@@ -1000,6 +1002,7 @@ def OMP_Flush : Directive<[Spelling<"flush">]> {
// TODO This should ne `none` instead. Comment carried over from
// OMPKinds.def.
VersionedClause<OMPC_Flush>,
+ VersionedClause<OMPC_Memscope, 60>,
VersionedClause<OMPC_Release, 50>,
VersionedClause<OMPC_SeqCst, 51>,
];
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
index 16fa4357b3b57f..ca275213fa702d 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
@@ -4092,10 +4092,12 @@ class OpenMPIRBuilder {
/// \param Loc The insert and source location description.
/// \param AO The required atomic ordering
/// \param AK The OpenMP atomic operation kind used.
+ /// \param Scope The memory scope of the flush.
///
/// \returns wether a flush was emitted or not
bool checkAndEmitFlushAfterAtomic(const LocationDescription &Loc,
- AtomicOrdering AO, AtomicKind AK);
+ AtomicOrdering AO, AtomicKind AK,
+ AtomicScope Scope = AtomicScope::Device);
/// Emit atomic update for constructs: X = X BinOp Expr ,or X = Expr BinOp X
/// For complex Operations: X = UpdateOp(X) => CmpExch X, old_X, UpdateOp(X)
@@ -4304,12 +4306,14 @@ class OpenMPIRBuilder {
const LocationDescription &Loc, AtomicOpValue &X, AtomicOpValue &V,
AtomicOpValue &R, Value *E, Value *D, AtomicOrdering AO,
omp::OMPAtomicCompareOp Op, bool IsXBinopExpr, bool IsPostfixUpdate,
- bool IsFailOnly, bool IsWeak = false);
+ bool IsFailOnly, bool IsWeak = false,
+ AtomicScope Scope = AtomicScope::Device);
LLVM_ABI InsertPointTy createAtomicCompare(
const LocationDescription &Loc, AtomicOpValue &X, AtomicOpValue &V,
AtomicOpValue &R, Value *E, Value *D, AtomicOrdering AO,
omp::OMPAtomicCompareOp Op, bool IsXBinopExpr, bool IsPostfixUpdate,
- bool IsFailOnly, AtomicOrdering Failure, bool IsWeak = false);
+ bool IsFailOnly, AtomicOrdering Failure, bool IsWeak = false,
+ AtomicScope Scope = AtomicScope::Device);
/// Create the control flow structure of a canonical OpenMP loop.
///
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index d765b1a3d28d74..a9350976490183 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -11567,7 +11567,8 @@ Error OpenMPIRBuilder::emitIfClause(Value *Cond, BodyGenCallbackTy ThenGen,
}
bool OpenMPIRBuilder::checkAndEmitFlushAfterAtomic(
- const LocationDescription &Loc, llvm::AtomicOrdering AO, AtomicKind AK) {
+ const LocationDescription &Loc, llvm::AtomicOrdering AO, AtomicKind AK,
+ AtomicScope Scope) {
assert(!(AO == AtomicOrdering::NotAtomic ||
AO == llvm::AtomicOrdering::Unordered) &&
"Unexpected Atomic Ordering.");
@@ -11599,7 +11600,7 @@ bool OpenMPIRBuilder::checkAndEmitFlushAfterAtomic(
// that belongs on entry, and a release fence there would order nothing. So
// the flush is seq_cst whichever flush the construct implies.
if (Flush)
- emitFlush(Loc);
+ emitFlush(Loc, AtomicOrdering::SequentiallyConsistent, Scope);
return Flush;
}
@@ -12004,18 +12005,19 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
const LocationDescription &Loc, AtomicOpValue &X, AtomicOpValue &V,
AtomicOpValue &R, Value *E, Value *D, AtomicOrdering AO,
omp::OMPAtomicCompareOp Op, bool IsXBinopExpr, bool IsPostfixUpdate,
- bool IsFailOnly, bool IsWeak) {
+ bool IsFailOnly, bool IsWeak, AtomicScope Scope) {
AtomicOrdering Failure = AtomicCmpXchgInst::getStrongestFailureOrdering(AO);
return createAtomicCompare(Loc, X, V, R, E, D, AO, Op, IsXBinopExpr,
- IsPostfixUpdate, IsFailOnly, Failure, IsWeak);
+ IsPostfixUpdate, IsFailOnly, Failure, IsWeak,
+ Scope);
}
OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
const LocationDescription &Loc, AtomicOpValue &X, AtomicOpValue &V,
AtomicOpValue &R, Value *E, Value *D, AtomicOrdering AO,
omp::OMPAtomicCompareOp Op, bool IsXBinopExpr, bool IsPostfixUpdate,
- bool IsFailOnly, AtomicOrdering Failure, bool IsWeak) {
+ bool IsFailOnly, AtomicOrdering Failure, bool IsWeak, AtomicScope Scope) {
if (!updateToLocation(Loc))
return Loc.IP;
@@ -12357,7 +12359,7 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
}
}
- checkAndEmitFlushAfterAtomic(Loc, AO, AtomicKind::Compare);
+ checkAndEmitFlushAfterAtomic(Loc, AO, AtomicKind::Compare, Scope);
return Builder.saveIP();
}
diff --git a/llvm/unittests/Frontend/OpenMPIRBuilderTest.cpp b/llvm/unittests/Frontend/OpenMPIRBuilderTest.cpp
index 1ec0f4c232a8bc..42358bb085086a 100644
--- a/llvm/unittests/Frontend/OpenMPIRBuilderTest.cpp
+++ b/llvm/unittests/Frontend/OpenMPIRBuilderTest.cpp
@@ -4313,6 +4313,37 @@ TEST_F(OpenMPIRBuilderTest, OMPAtomicCapture) {
EXPECT_FALSE(verifyModule(*M, &errs()));
}
+TEST_F(OpenMPIRBuilderTest, OMPAtomicCompareScope) {
+ // The flush after a seq_cst atomic compare uses the given scope.
+ M->setTargetTriple(Triple("nvptx64-nvidia-cuda"));
+ OpenMPIRBuilder OMPBuilder(*M);
+ OMPBuilder.Config.IsTargetDevice = true;
+ OMPBuilder.Config.setIsGPU(true);
+ OMPBuilder.initialize();
+ IRBuilder<> Builder(BB);
+
+ LLVMContext &Ctx = M->getContext();
+ IntegerType *Int32 = Type::getInt32Ty(Ctx);
+ AllocaInst *XVal = Builder.CreateAlloca(Int32);
+ OpenMPIRBuilder::AtomicOpValue X = {XVal, Int32, true, false};
+ OpenMPIRBuilder::AtomicOpValue V = {nullptr, nullptr, false, false};
+ OpenMPIRBuilder::AtomicOpValue R = {nullptr, nullptr, false, false};
+ ConstantInt *Expr = ConstantInt::get(Int32, 1U);
+ ConstantInt *D = ConstantInt::get(Int32, 2U);
+
+ Builder.restoreIP(OMPBuilder.createAtomicCompare(
+ Builder, X, V, R, Expr, D, AtomicOrdering::SequentiallyConsistent,
+ OMPAtomicCompareOp::EQ, true, false, false, /*IsWeak=*/false,
+ AtomicScope::Workgroup));
+ Builder.CreateRetVoid();
+ EXPECT_FALSE(verifyModule(*M, &errs()));
+
+ auto *Fence = dyn_cast<FenceInst>(BB->getTerminator()->getPrevNode());
+ ASSERT_NE(Fence, nullptr);
+ EXPECT_EQ(Fence->getOrdering(), AtomicOrdering::SequentiallyConsistent);
+ EXPECT_EQ(Fence->getSyncScopeID(), Ctx.getOrInsertSyncScopeID("block"));
+}
+
TEST_F(OpenMPIRBuilderTest, OMPAtomicCompare) {
OpenMPIRBuilder OMPBuilder(*M);
OMPBuilder.Config.IsTargetDevice = false;
More information about the llvm-branch-commits
mailing list