[all-commits] [llvm/llvm-project] 51aa83: [offload][OpenMP] Add atomic cross-team reductions

[Robert Imschweiler via All-commits]

  Branch: refs/heads/users/ro-i/xteam-red-atomic
  Home:   https://github.com/llvm/llvm-project
  Commit: 51aa8352dec89792a646259348eb17a002f6c035
      https://github.com/llvm/llvm-project/commit/51aa8352dec89792a646259348eb17a002f6c035
  Author: Robert Imschweiler <robert.imschweiler at amd.com>
  Date:   2026-07-13 (Mon, 13 Jul 2026)

  Changed paths:
    M clang/include/clang/Basic/LangOptions.def
    M clang/include/clang/Options/Options.td
    M clang/lib/CodeGen/CGOpenMPRuntimeGPU.cpp
    M clang/lib/Driver/ToolChains/Clang.cpp
    A clang/test/Driver/openmp-target-fast-reduction-flag.c
    A clang/test/OpenMP/target_teams_fast_reduction_codegen.cpp
    M llvm/include/llvm/Frontend/OpenMP/OMPKinds.def
    M llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
    M offload/test/offloading/multiple_reductions.cpp
    A offload/test/offloading/xteam_atomic_reduction_usm.cpp
    M openmp/device/include/Interface.h
    M openmp/device/src/Kernel.cpp

  Log Message:
  -----------
  [offload][OpenMP] Add atomic cross-team reductions

Regular cross-team reductions have two phases: the intra-team reduction
and the inter-team reduction. Atomic cross-team reductions replace the
second phase with a atomic instruction which is used by the main thread
of each team to directly fold the result of the intra-team reduction
into the final result. Since this requires a combination of "data type"
and "combine operation" for which an atomic instruction is available,
only some (but very common) reductions can be transformed to atomic
reductions. In cases where multiple reductions are performed on the same
construct, the atomic path is only taken if all reductions can be
transformed. Otherwise, we fall back to the regular cross-team reduction
using a buffer with per-team slots. This is not strictly necessary, but
hybrid reductions would induce more complexity with questionable
benefit.

Selecting an atomic path might not be the best option for every
situation, which is why it is not enabled by default. Instead, it can be
enabled via `-fopenmp-target-fast-reduction`. This flag has already been
established by ROCm for the same purpose and its wording leans on the
more general `-fopenmp-target-fast`. Note that enabling the atomic path
will not *force* atomic reductions. They will still only be applied if
possible, as described above.

The performance (measured with https://github.com/ro-i/xteam-test @
c71339705091500f731e2a39f247d2660bacbdce) is up to +15% faster (aka,
more throughput) for supported reductions on a gfx942, with no noticable
regressions.
Example:
- sum reduction, type double: +10.22% faster
- sum reduction, type uint: +15.57% faster
- sum reduction, type ulong: +13.31% faster

On a gfx90a, however, there is little to negative benefit:
- sum reduction, type double: -4.32% faster (aka, slower)
- sum reduction, type uint: +3.08% faster
- sum reduction, type ulong: +1.68% faster

Claude assisted with this patch.



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications