[llvm] [OpenMPIRBuilder] Emit a dispatch loop for ordered worksharing loops on the device (PR #214263)

Wed Aug 5 08:54:59 PDT 2026

https://github.com/sbryngelson created https://github.com/llvm/llvm-project/pull/214263

`ordered` inside a target region does not order anything in flang, so a conforming program gets
wrong results. Fixes #214257.

`applyWorkshareLoop` hands the loop to a `__kmpc_*_static_loop_*` entry on device and drops its
schedule-related arguments, `HasOrderedClause` among them. The device `__kmpc_ordered` is an empty
stub; the ordering actually comes from `__kmpc_dispatch_fini` in the dispatch loop, which is never
emitted. clang switches to dispatch when `ordered` is present, which is why the C equivalent works
on the same GPU.

Two changes:

* `applyWorkshareLoop` no longer takes the device shortcut when `ordered` is present, so the
  generic path emits a dispatch loop.
* `applyDynamicWorkshareLoop` casts the four bound allocas before passing them to
  `__kmpc_dispatch_next`. They are in the alloca address space, which is non-zero on AMDGPU, while
  the runtime entry takes generic pointers. Without this the device module fails verification with
  "Call parameter type does not match function signature". The casts fold away where the alloca
  address space is already zero, so the host path is unchanged. Loads and stores keep using the
  original allocas.

### Testing

New `offload/test/offloading/fortran/target-ordered.f90` records the order in which iterations
enter the ordered region.

| | before | after |
|---|---|---|
| the new test, gfx90a, 20 runs | 0 pass / 20 fail | **20 pass / 0 fail** |
| Fortran port of `offload/test/offloading/schedule.c`, 20 runs | 16 pass / 4 fail | **20 pass / 0 fail** |

Device IR for the ordered case goes from `__kmpc_for_static_loop_4u` to `__kmpc_dispatch_init_4u` /
`__kmpc_dispatch_next_4u` / `__kmpc_dispatch_fini_4u`. A loop without `ordered` still lowers to
`__kmpc_distribute_for_static_loop_4u`, unchanged.

`check-flang`, `mlir/test/Target/LLVMIR`, `mlir/test/Dialect/OpenMP`, `llvm/test/Frontend` and
`clang/test/OpenMP` all pass.

The `schedule` clause is dropped by the same line and is still ignored on device after this patch;
that is a separate issue and I will file it separately.

---

Parts of this change were written or audited with Claude Code. I have reviewed all of it and take
full responsibility for the contribution. See `llvm/docs/AIToolPolicy.md`.


>From c7ca80cd606734f913841fe83dc41ff0fb523e65 Mon Sep 17 00:00:00 2001
From: Spencer Bryngelson <sbryngelson at gmail.com>
Date: Wed, 5 Aug 2026 10:54:42 -0500
Subject: [PATCH] Emit a dispatch loop for ordered worksharing loops on the
 device.

Assisted-by: Claude
---
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     | 26 ++++++++++++++++---
 .../offloading/fortran/target-ordered.f90     | 26 +++++++++++++++++++
 2 files changed, 49 insertions(+), 3 deletions(-)
 create mode 100644 offload/test/offloading/fortran/target-ordered.f90

diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index ca191165c61c1..f6df028f4ecc2 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -6566,7 +6566,12 @@ OpenMPIRBuilder::InsertPointOrErrorTy OpenMPIRBuilder::applyWorkshareLoop(
     bool HasNonmonotonicModifier, bool HasOrderedClause,
     WorksharingLoopType LoopType, bool NoLoop, bool HasDistSchedule,
     Value *DistScheduleChunkSize) {
-  if (Config.isTargetDevice())
+  // The device path hands the loop to a __kmpc_*_static_loop_* entry, which
+  // cannot provide the ordering an `ordered` region needs: the device
+  // __kmpc_ordered is an empty stub and the ordering comes from
+  // __kmpc_dispatch_fini. Use the generic path so a dispatch loop is emitted,
+  // as clang does for the same construct.
+  if (Config.isTargetDevice() && !HasOrderedClause)
     return applyWorkshareLoopTarget(DL, CLI, AllocaIP, LoopType, NoLoop);
   OMPScheduleType EffectiveScheduleType = computeOpenMPScheduleType(
       SchedKind, ChunkSize, HasSimdModifier, HasMonotonicModifier,
@@ -6713,6 +6718,21 @@ OpenMPIRBuilder::applyDynamicWorkshareLoop(DebugLoc DL, CanonicalLoopInfo *CLI,
   Value *PStride = Builder.CreateAlloca(IVTy, nullptr, "p.stride");
   CLI->setLastIter(PLastIter);
 
+  // __kmpc_dispatch_next takes generic pointers. Where the alloca address
+  // space is not zero, as on AMDGPU, the allocas above are in the private
+  // address space and have to be cast for the call. The casts fold away when
+  // the alloca address space is already zero. Loads below keep using the
+  // original allocas.
+  PointerType *ArgPtrTy = Builder.getPtrTy();
+  Value *PLastIterArg =
+      Builder.CreatePointerBitCastOrAddrSpaceCast(PLastIter, ArgPtrTy);
+  Value *PLowerBoundArg =
+      Builder.CreatePointerBitCastOrAddrSpaceCast(PLowerBound, ArgPtrTy);
+  Value *PUpperBoundArg =
+      Builder.CreatePointerBitCastOrAddrSpaceCast(PUpperBound, ArgPtrTy);
+  Value *PStrideArg =
+      Builder.CreatePointerBitCastOrAddrSpaceCast(PStride, ArgPtrTy);
+
   // At the end of the preheader, prepare for calling the "init" function by
   // storing the current loop bounds into the allocated space. A canonical loop
   // always iterates from 0 to trip-count with step 1. Note that "init" expects
@@ -6755,8 +6775,8 @@ OpenMPIRBuilder::applyDynamicWorkshareLoop(DebugLoc DL, CanonicalLoopInfo *CLI,
   // This needs to be 32-bit always, so can't use the IVTy Zero above.
   Builder.SetInsertPoint(OuterCond, OuterCond->getFirstInsertionPt());
   Value *Res = createRuntimeFunctionCall(
-      DynamicNext,
-      {SrcLoc, ThreadNum, PLastIter, PLowerBound, PUpperBound, PStride});
+      DynamicNext, {SrcLoc, ThreadNum, PLastIterArg, PLowerBoundArg,
+                    PUpperBoundArg, PStrideArg});
   Constant *Zero32 = ConstantInt::get(I32Type, 0);
   Value *MoreWork = Builder.CreateCmp(CmpInst::ICMP_NE, Res, Zero32);
   Value *LowerBound =
diff --git a/offload/test/offloading/fortran/target-ordered.f90 b/offload/test/offloading/fortran/target-ordered.f90
new file mode 100644
index 0000000000000..b7e96bea5c833
--- /dev/null
+++ b/offload/test/offloading/fortran/target-ordered.f90
@@ -0,0 +1,26 @@
+! RUN: %libomptarget-compile-run-and-check-generic
+! REQUIRES: flang
+! REQUIRES: gpu
+
+! An ordered region must execute in iteration order. Record the order in which
+! iterations enter it; any inversion means the guarantee was not honoured.
+program ordered_target
+  implicit none
+  integer, parameter :: n = 64
+  integer :: seq(n), pos, i, bad
+  pos = 0
+  seq = -1
+  !$omp target parallel do ordered map(tofrom:seq,pos)
+  do i = 1, n
+     !$omp ordered
+     pos = pos + 1
+     seq(pos) = i
+     !$omp end ordered
+  end do
+  bad = 0
+  do i = 1, n
+     if (seq(i) /= i) bad = bad + 1
+  end do
+  ! CHECK: out of order: 0
+  print '(A,I4)', "out of order: ", bad
+end program

