[flang-commits] [clang] [flang] [flang][HLFIR] Add array-section reduction promotion HLFIR pass (PR #214511)
Caroline Newcombe via flang-commits
flang-commits at lists.llvm.org
Thu Aug 6 08:29:23 PDT 2026
https://github.com/cenewcombe created https://github.com/llvm/llvm-project/pull/214511
## Summary
This patch adds a new HLFIR-to-FIR pass, `array-section-reduction`, that recognizes a reduction into a **loop-invariant array section** and promotes the section to a constant-shape local temporary so LLVM's loop vectorizer can vectorize the loop. The pass is **opt-in** and **off by default**, gated behind a new experimental frontend flag `-fexperimental-array-section-reduction`.
Tracking issue: #208086.
RFC: Discourse thread — TBD.
## Motivation
This pass targets the force-accumulation kernel of the **SPEC OMP2012 350.md** molecular-dynamics benchmark, and the broader class of physics/engineering codes that reduce a small, fixed-size 3-vector / tensor into a loop-invariant slice of a larger array inside a hot loop. In 350.md the pattern is an O(n²) loop that accumulates a per-particle 3-vector force into `a(:,i)`:
```fortran
subroutine accumulate(a, x, i, n)
real(8) :: a(3,n), x(n) ! constant leading extent (the physical dimension)
integer :: i, n, j
a(:,i) = 0.0
do j = 1, n
a(:,i) = a(:,i) + f(x(j)) ! reduce a 3-vector into a loop-invariant section
end do
end subroutine
```
Even though `a(:,i)` is loop-invariant and tiny, flang lowers each in-loop `a(:,i) = a(:,i) + ...` as a store through the array descriptor. The loop vectorizer sees a may-alias memory dependence on that descriptor section and conservatively refuses to vectorize the loop, leaving the kernel scalar.
The classic manual fix is to reduce into a local fixed-size temporary and write it back after the loop. This pass performs that rewrite automatically when it is provably safe.
## What the pass does
For the pattern
```
a(:,i) = 0.0 ! (init) whole-section define
do j = ...
if (...) a(:,i) = a(:,i) + ... ! (RMW) invariant-section reduction
end do
... = f(a(:,i)) ! (use) post-loop consume
```
the pass promotes the section to a constant-shape local temporary `T`:
1. Allocate `T` with the section's constant shape and the **section's** element type.
2. Fold the dominating init into `T`.
3. Redirect the in-loop read-modify-write to `T`.
4. Copy `T` back to the original section after the loop.
5. At `-O2`/`-O3`, annotate the reduction loop with `llvm.loop.vectorize.enable` (see options below).
This removes the memory dependence on the loop-invariant descriptor section and turns the reduction into a fixed-size register-scalarizable accumulation.
## Safety / preconditions
Promotion only fires when all of the following hold (see the file-level comment in `ArraySectionReduction.cpp` for details):
- **P0** — the section element type is a trivial, non-volatile value type (numeric/logical). Character, derived, polymorphic, and volatile sections are rejected.
- **P1** — the section address is loop-invariant (no in-loop store or call rewrites the memory its subscripts read).
- **P2** — an unconditional store to the identical whole section dominates the loop, and nothing between that init and the loop touches the section.
- **P3** — every other in-loop access to the section is redirected to `T`: no aliasing write may bypass it, no read may observe the un-updated section, and no call may touch its storage. Aliasing is checked with `fir::AliasAnalysis`, failing closed on unresolved effects and opaque side-effecting ops.
- **P4** — the reduction has a compile-time-constant shape whose element count is at most `max-reduction-extent`; larger or runtime-shaped sections are detected but left unchanged.
## Pass options (TableGen)
- `force-vectorize` (bool, default `true`) — annotate the promoted loop with `llvm.loop.vectorize.enable`. The promoted temporary has a compile-time constant shape, so it scalarizes into registers and the loop becomes a plain reduction. Disable to leave the vectorization decision to the cost model.
- `max-reduction-extent` (unsigned, default `8`) — only promote sections whose constant element count is at most this. Each element becomes a reduction accumulator kept in a vector register across the loop, so a too-large count exhausts the register file and spills. `8` is a conservative, target-independent default (8 f64 accumulators occupy 4 of 16 vector registers on a 128-bit target).
- `max-unrollable-trip-count` (unsigned, default `64`) — a constant-bound inner loop with at most this trip count is assumed to be fully unrolled before the vectorizer runs, so it does not block annotating the enclosing reduction loop. This only affects whether the vectorization hint is emitted, never correctness.
## Driver / pipeline wiring
- Off by default. Enabled with `-fexperimental-array-section-reduction` (`-fno-experimental-array-section-reduction` to disable).
- Scheduled in `createHLFIRToFIRPassPipeline` at optimization level `O1`+, before `OptimizedBufferization`/`InlineHLFIRAssign` (while the reduction is still a single `hlfir.assign` of a designate). The vectorization hint is only requested at `O2`/`O3`, where the LLVM loop vectorizer runs; the promotion itself still applies at `O1`.
- Wiring: `CodeGenOptions.def`, `FlangOptions.td`, `CompilerInvocation.cpp`, `CrossToolHelpers.h` (`MLIRToLLVMPassPipelineConfig::ArraySectionReduction`), and a `config.ArraySectionReduction` gate in `Pipelines.cpp`. No `-O` level auto-enables it.
## Testing
- `flang/test/HLFIR/array-section-reduction.fir` — the core unit tests: one positive case per promotion path and one negative case per precondition (partial slice, missing/conditional init, aliasing read/write, in-loop call, non-invariant index, runtime shape, derived/volatile element type, read/write/copy between init and loop, direct `fir.array_coor` + `fir.load`/`fir.store`, `fir.copy`, `hlfir.assign` copy, CFG/OpenMP/nested variants, and vectorize-hint gating).
- `flang/test/HLFIR/array-section-reduction-extent.fir` — the `max-reduction-extent` boundary (accepted at the default, rejected at boundary+1, promoted when the option is raised).
- `flang/test/Driver/array-section-reduction.f90` — flag forwarding/precedence and pipeline insertion (verified by dumping the MLIR pass pipeline, since an MLIR pass is not visible to LLVM's `-print-pipeline-passes`).
## Performance
Measured on SPEC OMP2012 350.md, AMD EPYC 7663, 64 threads:
- **~2.5x** whole-benchmark speedup from this pass alone.
- **~8x** whole-benchmark speedup with all PRs linked to #208086 applied together.
Note: a vectorized reduction loop only yields an end-to-end win when the loop body itself vectorizes; for kernels that call transcendental math this requires a vector math library (e.g. `-fveclib=libmvec`). This pass makes the loop legal and, optionally, forced for vectorization; the vector math library supplies the arithmetic throughput.
## Caveats and future work
- **Plan to default.** The intent is to make this pass default *if* a broader benchmarking investigation (across suites and targets) supports it; until then it is opt-in behind the experimental flag.
- **`force-vectorize=true` overrides the cost model.** By default the pass emits `llvm.loop.vectorize.enable` on the promoted loop at `-O2`/`-O3`, overriding the LLVM loop vectorizer's cost model for that loop. This is intentional — the promoted accumulator is constant-shape and register-scalarizable, a shape the cost model tends to under-value — but a mispredicted case could vectorize unprofitably. `force-vectorize=false` promotes without the hint and leaves the decision to the cost model.
- **Constant-extent requirement (P4).** The section's extent must be a compile-time constant. Allocatable / assumed-shape sections with runtime extents do not qualify without a source change giving the leading (physical) dimension a constant bound.
- **`max-reduction-extent`** is a target-independent stand-in framed as forced-vectorization *profitability* (register pressure), not as a stack-size policy — it runs upstream of all allocation-placement passes, so it makes no stack-vs-heap decision. The default `8` is hand-picked; the principled bound would come from `TargetTransformInfo` (register count / preferred VF), which HLFIR passes do not currently plumb (noted as a TODO in the pass).
- **Scope.** Only the single-dominating-init, whole-section, single-RMW shape is recognized; more general reduction shapes are left for future work. `max-unrollable-trip-count` (64) is a hint-only heuristic with no correctness impact, and section-equivalence is fail-closed on commutativity (missed optimizations only).
**Assisted by**: GitHub Copilot.
>From 3e8489abc10554bb4eb2fbbe4c7d079e614068e0 Mon Sep 17 00:00:00 2001
From: Caroline Newcombe <caroline.newcombe at hpe.com>
Date: Thu, 30 Jul 2026 14:02:01 -0500
Subject: [PATCH] [flang] Add array-section reduction promotion HLFIR pass
Add an opt-in HLFIR pass (-farray-section-reduction) that promotes a
loop-invariant array-section reduction such as
a(:,i) = 0.0
do j = ...
a(:,i) = a(:,i) + x(:)
end do
to a constant-shape local temporary that is written back after the loop.
This removes the in-loop store to the loop-invariant descriptor section
that blocks LLVM's loop vectorizer, and (at O2/O3) annotates the loop
with llvm.loop.vectorize.enable.
Promotion requires a set of preconditions -- trivial non-volatile
element type, loop-invariant section address, a dominating whole-section
init, no other aliasing access in the loop, and a compile-time-constant
shape within max-reduction-extent -- checked via structural
section-equivalence, dominance, and fir::AliasAnalysis.
Assisted-by: Github Copilot
---
clang/include/clang/Options/FlangOptions.td | 4 +
clang/lib/Driver/ToolChains/Flang.cpp | 2 +
.../include/flang/Frontend/CodeGenOptions.def | 1 +
flang/include/flang/Optimizer/HLFIR/Passes.td | 33 +
flang/include/flang/Tools/CrossToolHelpers.h | 2 +
flang/lib/Frontend/CompilerInvocation.cpp | 5 +
.../Transforms/ArraySectionReduction.cpp | 777 +++++++
.../Optimizer/HLFIR/Transforms/CMakeLists.txt | 1 +
flang/lib/Optimizer/Passes/Pipelines.cpp | 11 +
flang/test/Driver/array-section-reduction.f90 | 46 +
.../HLFIR/array-section-reduction-extent.fir | 86 +
flang/test/HLFIR/array-section-reduction.fir | 1896 +++++++++++++++++
12 files changed, 2864 insertions(+)
create mode 100644 flang/lib/Optimizer/HLFIR/Transforms/ArraySectionReduction.cpp
create mode 100644 flang/test/Driver/array-section-reduction.f90
create mode 100644 flang/test/HLFIR/array-section-reduction-extent.fir
create mode 100644 flang/test/HLFIR/array-section-reduction.fir
diff --git a/clang/include/clang/Options/FlangOptions.td b/clang/include/clang/Options/FlangOptions.td
index 6b375c3b2b7dc..92902b453a715 100644
--- a/clang/include/clang/Options/FlangOptions.td
+++ b/clang/include/clang/Options/FlangOptions.td
@@ -265,6 +265,10 @@ defm loop_versioning : BoolOptionWithoutMarshalling<"f", "version-loops-for-stri
PosFlag<SetTrue, [], [ClangOption], "Create unit-strided versions of loops">,
NegFlag<SetFalse, [], [ClangOption], "Do not create unit-strided loops (default)">>;
+defm experimental_array_section_reduction : BoolOptionWithoutMarshalling<"f", "experimental-array-section-reduction",
+ PosFlag<SetTrue, [], [FlangOption], "Promote loop-invariant array-section reductions to a local temporary and enable their vectorization (experimental)">,
+ NegFlag<SetFalse, [], [FlangOption], "Do not promote array-section reductions (default)">>;
+
defm stack_repack_arrays
: BoolOptionWithoutMarshalling<
"f", "stack-repack-arrays",
diff --git a/clang/lib/Driver/ToolChains/Flang.cpp b/clang/lib/Driver/ToolChains/Flang.cpp
index 25bb9832432cc..4f2209de5d153 100644
--- a/clang/lib/Driver/ToolChains/Flang.cpp
+++ b/clang/lib/Driver/ToolChains/Flang.cpp
@@ -339,6 +339,8 @@ void Flang::addCodegenOptions(const ArgList &Args,
options::OPT_fno_experimental_loop_fusion);
Args.addOptInFlag(CmdArgs, options::OPT_freal_sum_reassociation,
options::OPT_fno_real_sum_reassociation);
+ Args.addOptInFlag(CmdArgs, options::OPT_fexperimental_array_section_reduction,
+ options::OPT_fno_experimental_array_section_reduction);
handleInterchangeLoopsArgs(Args, CmdArgs);
handleVectorizeLoopsArgs(Args, CmdArgs);
diff --git a/flang/include/flang/Frontend/CodeGenOptions.def b/flang/include/flang/Frontend/CodeGenOptions.def
index d49a7f3647eec..5e85d62059019 100644
--- a/flang/include/flang/Frontend/CodeGenOptions.def
+++ b/flang/include/flang/Frontend/CodeGenOptions.def
@@ -54,6 +54,7 @@ CODEGENOPT(VectorizeSLP, 1, 0) ///< Enable SLP vectorization.
CODEGENOPT(InterchangeLoops, 1, 0) ///< Enable loop interchange.
CODEGENOPT(FuseLoops, 1, 0) ///< Enable loop fusion.
CODEGENOPT(LoopVersioning, 1, 0) ///< Enable loop versioning.
+CODEGENOPT(ArraySectionReduction, 1, 0) ///< -fexperimental-array-section-reduction (promote loop-invariant array-section reductions)
CODEGENOPT(SplitSumExpressionTree, 1, 0) ///< Split REAL addition expression trees.
CODEGENOPT(UnrollLoops, 1, 0) ///< Enable loop unrolling
CODEGENOPT(AliasAnalysis, 1, 0) ///< Enable alias analysis pass
diff --git a/flang/include/flang/Optimizer/HLFIR/Passes.td b/flang/include/flang/Optimizer/HLFIR/Passes.td
index 2d57a50acb304..c516b60ad03d5 100644
--- a/flang/include/flang/Optimizer/HLFIR/Passes.td
+++ b/flang/include/flang/Optimizer/HLFIR/Passes.td
@@ -112,4 +112,37 @@ def PropagateFortranVariableAttributes : Pass<"propagate-fortran-attrs"> {
let summary = "Propagate FortranVariableFlagsAttr attributes through HLFIR";
}
+def ArraySectionReduction : Pass<"array-section-reduction"> {
+ let summary = "Promote loop-invariant array-section reductions to a local temporary";
+ let description = [{
+ Detect a loop-carried reduction into a loop-invariant array section that is
+ unconditionally defined before the loop (e.g. `a(:,i) = a(:,i) + x(:)`), and
+ promote the section to a constant-shape local temporary that is written back
+ after the loop. The loop is then annotated to enable vectorization: promoting
+ the section removes the memory dependence on the loop-invariant descriptor
+ that would otherwise block the loop vectorizer.
+ }];
+ let options = [
+ Option<"forceVectorize", "force-vectorize", "bool", /*default=*/"true",
+ "Annotate the promoted loop with llvm.loop.vectorize.enable to "
+ "override the cost model. The promoted temporary has a "
+ "compile-time constant shape, so it scalarizes into individual "
+ "registers and the loop becomes a plain reduction; disable to "
+ "leave the vectorization decision to the cost model.">,
+ Option<"maxReductionExtent", "max-reduction-extent", "unsigned",
+ /*default=*/"8",
+ "Only promote sections whose constant element count is at most "
+ "this. Each element becomes a reduction accumulator kept in a "
+ "vector register across the loop, so a too-large count exhausts "
+ "the registers and spills.">,
+ Option<"maxUnrollableTripCount", "max-unrollable-trip-count", "unsigned",
+ /*default=*/"64",
+ "A constant-bound inner loop with at most this trip count is "
+ "assumed to be fully unrolled before the vectorizer runs, so it "
+ "does not block annotating the enclosing reduction loop. This "
+ "only affects whether the vectorization hint is emitted, never "
+ "correctness.">];
+ let dependentDialects = ["mlir::LLVM::LLVMDialect"];
+}
+
#endif //FORTRAN_DIALECT_HLFIR_PASSES
diff --git a/flang/include/flang/Tools/CrossToolHelpers.h b/flang/include/flang/Tools/CrossToolHelpers.h
index 6569d34e0f255..a61650fbd1b40 100644
--- a/flang/include/flang/Tools/CrossToolHelpers.h
+++ b/flang/include/flang/Tools/CrossToolHelpers.h
@@ -123,6 +123,7 @@ struct MLIRToLLVMPassPipelineConfig : public FlangEPCallBacks {
EnableSafeTrampoline = opts.EnableSafeTrampoline;
Underscoring = opts.Underscoring;
LoopVersioning = opts.LoopVersioning;
+ ArraySectionReduction = opts.ArraySectionReduction;
DebugInfo = opts.getDebugInfo();
AliasAnalysis = opts.AliasAnalysis;
FramePointerKind = opts.getFramePointer();
@@ -153,6 +154,7 @@ struct MLIRToLLVMPassPipelineConfig : public FlangEPCallBacks {
bool EnableSafeTrampoline{false}; ///< Use runtime trampoline pool (W^X).
bool Underscoring = true; ///< add underscores to function names.
bool LoopVersioning = false; ///< Run the version loop pass.
+ bool ArraySectionReduction = false; ///< Promote array-section reductions.
bool AliasAnalysis = false; ///< Add TBAA tags to generated LLVMIR.
llvm::codegenoptions::DebugInfoKind DebugInfo =
llvm::codegenoptions::NoDebugInfo; ///< Debug info generation.
diff --git a/flang/lib/Frontend/CompilerInvocation.cpp b/flang/lib/Frontend/CompilerInvocation.cpp
index 33ce6cb6869f1..68fb566896f8e 100644
--- a/flang/lib/Frontend/CompilerInvocation.cpp
+++ b/flang/lib/Frontend/CompilerInvocation.cpp
@@ -336,6 +336,11 @@ static void parseCodeGenArgs(Fortran::frontend::CodeGenOptions &opts,
clang::options::OPT_fno_loop_versioning, false))
opts.LoopVersioning = 1;
+ if (args.hasFlag(clang::options::OPT_fexperimental_array_section_reduction,
+ clang::options::OPT_fno_experimental_array_section_reduction,
+ false))
+ opts.ArraySectionReduction = 1;
+
opts.UnrollLoops = args.hasFlag(clang::options::OPT_funroll_loops,
clang::options::OPT_fno_unroll_loops,
(opts.OptimizationLevel > 1));
diff --git a/flang/lib/Optimizer/HLFIR/Transforms/ArraySectionReduction.cpp b/flang/lib/Optimizer/HLFIR/Transforms/ArraySectionReduction.cpp
new file mode 100644
index 0000000000000..a08b7d637a0a3
--- /dev/null
+++ b/flang/lib/Optimizer/HLFIR/Transforms/ArraySectionReduction.cpp
@@ -0,0 +1,777 @@
+//===- ArraySectionReduction.cpp - Promote array-section reductions -------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Pass implementing the "loop-invariant array-section reduction promotion"
+// transform.
+//
+// It looks for the pattern:
+//
+// a(:,i) = 0.0 ! (init) whole-section define
+// do j = ...
+// if (...) a(:,i) = a(:,i) + ... ! (RMW) invariant-section reduction
+// end do
+// ... = f(a(:,i)) ! (use) post-loop consume
+//
+// When preconditions hold, it promotes the section to a constant-shape local
+// temporary T: fold the init into T, redirect the in-loop read-modify-write to
+// T, and copy T back to the section after the loop (see promote()). This turns
+// the in-loop store to a loop-invariant descriptor section into a reduction
+// into a local fixed-size array, removing the memory dependence that blocks
+// LLVM's loop vectorizer.
+//
+// Preconditions checked:
+// P0 the section element type is a trivial value type (numeric or logical)
+// and non-volatile; character, derived, polymorphic, and volatile
+// sections are not handled,
+// P1 the section address is loop-invariant in the enclosing loop (no store
+// or call inside the loop rewrites the memory its subscripts read),
+// P2 an unconditional store to the identical whole section dominates the
+// loop (proven by structural section-equivalence + dominance),
+// P3 every other access to the section inside the loop is redirected to the
+// temporary: no write may alias it, no read may bypass it, and no call
+// may modify or read its storage (fir::AliasAnalysis),
+// P4 the RHS carries a compile-time-constant shape whose extent is small
+// enough to scalarize into register accumulators; larger or
+// runtime-shaped sections are detected but not rewritten.
+//
+//===----------------------------------------------------------------------===//
+
+#include "flang/Optimizer/Analysis/AliasAnalysis.h"
+#include "flang/Optimizer/Builder/FIRBuilder.h"
+#include "flang/Optimizer/Builder/HLFIRTools.h"
+#include "flang/Optimizer/Dialect/FIROps.h"
+#include "flang/Optimizer/Dialect/FIRType.h"
+#include "flang/Optimizer/HLFIR/HLFIROps.h"
+#include "flang/Optimizer/HLFIR/Passes.h"
+#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/Utils/StaticValueUtils.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/OperationSupport.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Pass/Pass.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SmallPtrSet.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Support/MathExtras.h"
+
+namespace hlfir {
+#define GEN_PASS_DEF_ARRAYSECTIONREDUCTION
+#include "flang/Optimizer/HLFIR/Passes.h.inc"
+} // namespace hlfir
+
+#define DEBUG_TYPE "array-section-reduction"
+
+namespace {
+
+/// Strip no-op address/value forwarding to get at the underlying SSA value.
+static mlir::Value stripConvert(mlir::Value v) {
+ while (v) {
+ if (auto conv = v.getDefiningOp<fir::ConvertOp>()) {
+ v = conv.getValue();
+ continue;
+ }
+ break;
+ }
+ return v;
+}
+
+/// Strip hlfir.declare / fir.declare so two references to the same declared
+/// entity compare equal even when reached through different declare results.
+static mlir::Value stripDeclare(mlir::Value v) {
+ v = stripConvert(v);
+ if (auto res = mlir::dyn_cast<mlir::OpResult>(v)) {
+ mlir::Operation *def = res.getOwner();
+ if (auto d = mlir::dyn_cast<hlfir::DeclareOp>(def))
+ return d.getMemref();
+ if (auto d = mlir::dyn_cast<fir::DeclareOp>(def))
+ return d.getMemref();
+ }
+ return v;
+}
+
+/// Structural value-equivalence: true when \p a and \p b provably evaluate to
+/// the same runtime value. Handles the address/shape computations the front end
+/// emits for section designators (loads, box_dims, designates, shapes, arith).
+/// \p depth caps the recursion into operand trees; exceeding it fails closed
+/// (no match -> no promotion), which is always sound.
+static constexpr unsigned maxEquivRecursionDepth = 32;
+static bool equiv(mlir::Value a, mlir::Value b, unsigned depth = 0) {
+ if (depth > maxEquivRecursionDepth) // guard pathologically nested designators
+ return false;
+ a = stripConvert(a);
+ b = stripConvert(b);
+ if (!a || !b)
+ return false;
+ if (a == b)
+ return true;
+
+ // Two loads match by their declared address, ignoring any store between them.
+ // Sound only because the caller proves the loaded value is unchanged between
+ // the two program points: P1 (no in-loop write to the address deps) and the
+ // P2 interval check (section and subscripts untouched from init to loop).
+ auto la = a.getDefiningOp<fir::LoadOp>();
+ auto lb = b.getDefiningOp<fir::LoadOp>();
+ if (la || lb)
+ return la && lb &&
+ stripDeclare(la.getMemref()) == stripDeclare(lb.getMemref());
+
+ auto ra = mlir::dyn_cast<mlir::OpResult>(a);
+ auto rb = mlir::dyn_cast<mlir::OpResult>(b);
+ if (!ra || !rb || ra.getResultNumber() != rb.getResultNumber())
+ return false;
+
+ // Only match address/shape/index computations. hlfir.designate and
+ // fir.box_dims are pure address/descriptor-inspection ops, so allow them.
+ auto isAddressOrShapeOp = [](mlir::Operation *op) {
+ return mlir::isMemoryEffectFree(op) ||
+ mlir::isa<fir::BoxDimsOp, fir::ShapeOp, fir::ShapeShiftOp,
+ hlfir::DesignateOp>(op);
+ };
+ if (!isAddressOrShapeOp(ra.getOwner()) || !isAddressOrShapeOp(rb.getOwner()))
+ return false;
+
+ // Structural match only: commutativity is not modeled (i+1 vs 1+i fails to
+ // match), which costs a missed promotion but never a wrong one.
+ return mlir::OperationEquivalence::isEquivalentTo(
+ ra.getOwner(), rb.getOwner(),
+ /*checkEquivalent=*/
+ [depth](mlir::Value oa, mlir::Value ob) {
+ return mlir::success(equiv(oa, ob, depth + 1));
+ },
+ /*markEquivalent=*/nullptr,
+ mlir::OperationEquivalence::Flags::IgnoreLocations);
+}
+
+/// True when two hlfir.designate results address the identical array section.
+static bool sameSection(hlfir::DesignateOp d1, hlfir::DesignateOp d2) {
+ return equiv(d1.getResult(), d2.getResult());
+}
+
+/// Collect the hlfir.designate reads of the same section as \p lhsDes anywhere
+/// in the expression producing \p rhs (including nested hlfir.elemental ops).
+/// Matched by section identity, not SSA id. True if any read was found (RMW).
+static bool
+collectSectionReads(mlir::Value rhs, hlfir::DesignateOp lhsDes,
+ llvm::SmallVectorImpl<hlfir::DesignateOp> &reads) {
+ mlir::Operation *rhsDef = rhs.getDefiningOp();
+ if (!rhsDef)
+ return false;
+ llvm::SmallPtrSet<mlir::Operation *, 4> seen;
+ rhsDef->walk([&](mlir::Operation *op) {
+ for (mlir::Value operand : op->getOperands())
+ if (auto d = operand.getDefiningOp<hlfir::DesignateOp>())
+ if (sameSection(d, lhsDes) && seen.insert(d).second)
+ reads.push_back(d);
+ });
+ return !reads.empty();
+}
+
+/// Collect the declared addresses a section designator loads to form its
+/// address (e.g. the index i in a(:,i)), by walking its operand cone for loads.
+static llvm::SmallVector<mlir::Value>
+collectAddressDeps(hlfir::DesignateOp lhsDes) {
+ llvm::SmallVector<mlir::Value> addrDeps;
+ llvm::SmallPtrSet<mlir::Value, 8> seenDeps;
+ llvm::SmallPtrSet<mlir::Operation *, 16> visited;
+ llvm::SmallVector<mlir::Value, 16> worklist;
+ for (mlir::Value operand : lhsDes.getOperation()->getOperands())
+ worklist.push_back(operand);
+ while (!worklist.empty()) {
+ mlir::Operation *def = worklist.pop_back_val().getDefiningOp();
+ if (!def || !visited.insert(def).second)
+ continue;
+ if (auto load = mlir::dyn_cast<fir::LoadOp>(def)) {
+ // Keep the [hl]fir.declare address: AliasAnalysis classifies the variable
+ // (e.g. an OpenMP-private index) by walking through its declare, so
+ // stripping it here would defeat that and force a conservative MayAlias.
+ mlir::Value m = load.getMemref();
+ if (seenDeps.insert(m).second)
+ addrDeps.push_back(m);
+ }
+ for (mlir::Value operand : def->getOperands())
+ worklist.push_back(operand);
+ }
+ return addrDeps;
+}
+
+/// True when \p loop has constant bounds and a trip count at most \p
+/// maxTripCount -- the threshold below which we assume the unroller fully
+/// unrolls it before the vectorizer runs, so a nested loop no longer blocks
+/// annotating the enclosing reduction. This only gates the separable
+/// vectorization hint: an over-estimate at most annotates a loop the vectorizer
+/// then ignores (a surviving inner loop keeps the outer non-innermost), and an
+/// under-estimate omits a hint it could have used -- neither is a correctness
+/// change nor a regression.
+/// TODO: a target-aware threshold would come from TargetTransformInfo.
+static bool isFullyUnrollableLoop(fir::DoLoopOp loop, int64_t maxTripCount) {
+ std::optional<int64_t> lb = mlir::getConstantIntValue(loop.getLowerBound());
+ std::optional<int64_t> ub = mlir::getConstantIntValue(loop.getUpperBound());
+ std::optional<int64_t> step = mlir::getConstantIntValue(loop.getStep());
+ if (!lb || !ub || !step || *step <= 0)
+ return false;
+ // Overflow here means an astronomically large loop, i.e. not unrollable.
+ int64_t range, numerator;
+ if (llvm::SubOverflow(*ub, *lb, range) ||
+ llvm::AddOverflow(range, *step, numerator))
+ return false;
+ int64_t trip = numerator > 0 ? numerator / *step : 0;
+ return trip <= maxTripCount;
+}
+
+class ArraySectionReductionPass
+ : public hlfir::impl::ArraySectionReductionBase<ArraySectionReductionPass> {
+public:
+ using ArraySectionReductionBase<
+ ArraySectionReductionPass>::ArraySectionReductionBase;
+
+ void runOnOperation() override {
+ mlir::Operation *root = getOperation();
+
+ // TODO: only fir.do_loop is handled. Extend to other loop carriers
+ // (fir.iterate_while, and the OpenMP loop nest) to cover more kernels.
+ llvm::SmallVector<fir::DoLoopOp> loops;
+ root->walk([&](fir::DoLoopOp loop) { loops.push_back(loop); });
+ if (loops.empty())
+ return; // no loop to promote into; skip the dominance computation
+
+ // Op-agnostic, like the sibling HLFIR passes: the pipeline runs it per
+ // top-level op, so root is usually a func; under fir-opt it may be the
+ // module. That stays correct because every dominance query is intra-region
+ // (findDominatingFullDef stops at root, never crossing functions) and
+ // DominanceInfo is computed lazily per region.
+ mlir::DominanceInfo domInfo(root);
+ fir::AliasAnalysis aliasAnalysis;
+ // domInfo stays valid across the promotions below: promote() only inserts
+ // ops and rewrites operands, never changing the CFG (no block add/remove).
+ for (fir::DoLoopOp loop : loops)
+ matchLoop(loop, domInfo, aliasAnalysis);
+ }
+
+private:
+ /// Promote every loop-invariant array-section reduction found in \p loop.
+ void matchLoop(fir::DoLoopOp loop, mlir::DominanceInfo &domInfo,
+ fir::AliasAnalysis &aliasAnalysis) {
+ struct Candidate {
+ hlfir::AssignOp init;
+ hlfir::AssignOp rmw;
+ llvm::SmallVector<hlfir::DesignateOp> reads;
+ };
+ llvm::SmallVector<Candidate> candidates;
+
+ loop.walk([&](hlfir::AssignOp rmw) {
+ hlfir::Entity lhs(rmw.getLhs());
+ if (!lhs.isArray())
+ return;
+ auto lhsDes = lhs.getDefiningOp<hlfir::DesignateOp>();
+ if (!lhsDes)
+ return;
+ LLVM_DEBUG(llvm::dbgs()
+ << "array-section-reduction: candidate array-section assign "
+ << rmw.getLoc() << "\n");
+
+ // RMW reduction: the RHS must read the same section it writes.
+ llvm::SmallVector<hlfir::DesignateOp> reads;
+ if (!collectSectionReads(rmw.getRhs(), lhsDes, reads)) {
+ LLVM_DEBUG(llvm::dbgs()
+ << " reject: RHS does not read the section (not an RMW)\n");
+ return;
+ }
+
+ // P0: the section element type must be a non-volatile trivial value
+ // type.
+ mlir::Type eleTy = lhs.getFortranElementType();
+ if (!fir::isa_trivial(eleTy)) {
+ LLVM_DEBUG(llvm::dbgs()
+ << " reject: non-trivial section element type (P0)\n");
+ return;
+ }
+ if (fir::isa_volatile_type(lhs.getType())) {
+ LLVM_DEBUG(llvm::dbgs() << " reject: volatile section (P0)\n");
+ return;
+ }
+
+ // P4: the RHS must have a compile-time-constant shape whose extent is
+ // small enough to scalarize into register accumulators.
+ // maxReductionExtent (the max-reduction-extent option) tracks SIMD
+ // register pressure.
+ // TODO: derive the bound from TargetTransformInfo when it is available
+ // to this pass instead of a fixed default.
+ std::optional<int64_t> extent = rhsConstantExtent(rmw.getRhs());
+ if (!extent || *extent > maxReductionExtent) {
+ LLVM_DEBUG(llvm::dbgs()
+ << " reject: runtime or too-large section shape (P4)\n");
+ return;
+ }
+
+ // P2: an unconditional store to the identical whole section dominating L.
+ hlfir::AssignOp init = findDominatingFullDef(loop, lhsDes, domInfo);
+ if (!init) {
+ LLVM_DEBUG(llvm::dbgs()
+ << " reject: no dominating full-section init (P2)\n");
+ return;
+ }
+
+ // Address dependences of the section (subscripts it loads, e.g. i).
+ llvm::SmallVector<mlir::Value> addrDeps = collectAddressDeps(lhsDes);
+
+ // P1: the section address must be loop-invariant.
+ if (!sectionAddressLoopInvariant(loop, addrDeps, aliasAnalysis)) {
+ LLVM_DEBUG(llvm::dbgs()
+ << " reject: section address not loop-invariant (P1)\n");
+ return;
+ }
+
+ // P2 (interval): nothing may access the section between the init and the
+ // loop.
+ if (!sectionUntouchedBetweenInitAndLoop(init, loop, lhs, addrDeps,
+ domInfo, aliasAnalysis)) {
+ LLVM_DEBUG(
+ llvm::dbgs()
+ << " reject: section accessed between init and loop (P2)\n");
+ return;
+ }
+
+ // P3 (writes): no other write, and no call touching the section, inside
+ // L.
+ if (!noAliasingWritesInLoop(loop, rmw, lhs, aliasAnalysis)) {
+ LLVM_DEBUG(llvm::dbgs() << " reject: possible aliasing write or call "
+ "inside the loop (P3)\n");
+ return;
+ }
+
+ // P3 (reads): every in-loop access to the section must be redirected to
+ // T.
+ if (!allAliasingAccessesRedirected(loop, lhs, lhsDes, reads,
+ aliasAnalysis)) {
+ LLVM_DEBUG(llvm::dbgs()
+ << " reject: section accessed in the loop outside the "
+ "read-modify-write (P3)\n");
+ return;
+ }
+
+ LLVM_DEBUG(llvm::dbgs()
+ << " loop-invariant array-section reduction; init dominates "
+ "at "
+ << init.getLoc() << "\n");
+ candidates.push_back({init, rmw, std::move(reads)});
+ });
+
+ for (Candidate &c : candidates)
+ promote(loop, c.init, c.rmw, c.reads);
+ }
+
+ /// Rewrite the matched reduction to accumulate into a constant-shape local
+ /// temporary T (steps below), removing the in-loop store to the
+ /// loop-invariant descriptor section that blocks LLVM's loop vectorizer.
+ void promote(fir::DoLoopOp loop, hlfir::AssignOp init, hlfir::AssignOp rmw,
+ llvm::ArrayRef<hlfir::DesignateOp> reads) {
+ mlir::Location loc = rmw.getLoc();
+ // RHS is a constant-shape array expr (P4); use its shape for T.
+ auto rhsSeqTy = mlir::cast<fir::SequenceType>(
+ hlfir::getFortranElementOrSequenceType(rmw.getRhs().getType()));
+ mlir::Type seqTy = fir::SequenceType::get(
+ rhsSeqTy.getShape(),
+ hlfir::Entity(rmw.getLhs()).getFortranElementType());
+
+ // 1. Allocate T. createTemporary hoists the constant-shape alloca to the
+ // enclosing alloca block, so it is allocated once and stays
+ // thread-local inside an outlined OpenMP region.
+ mlir::OpBuilder opBuilder(init);
+ fir::FirOpBuilder builder(opBuilder, init.getOperation());
+ mlir::Value temp = builder.createTemporary(loc, seqTy, ".array_reduction");
+
+ // Capture the section (e.g. a(:,i)) before step 2 retargets the init;
+ // reused as the copy-out target in step 4.
+ mlir::Value section = init.getLhs();
+
+ // 2. Fold the init into T.
+ init.getLhsMutable().assign(temp);
+
+ // 3. Redirect the in-loop RMW to read and write T.
+ for (hlfir::DesignateOp read : reads)
+ read.getResult().replaceUsesWithIf(temp, [&](mlir::OpOperand &use) {
+ return loop->isProperAncestor(use.getOwner());
+ });
+ rmw.getLhsMutable().assign(temp);
+
+ // 4. Copy T back to the section after the loop (a(:,i) = T).
+ builder.setInsertionPointAfter(loop);
+ hlfir::AssignOp::create(builder, loc, temp, section, /*realloc=*/false,
+ /*keep_lhs_length_if_realloc=*/false,
+ /*temporary_lhs=*/false);
+
+ // 5. Force-enable vectorization: once T SROAs into scalar accumulators the
+ // loop is a plain reduction, so overriding the cost model is intended.
+ // Only requested at O2/O3; opt out with -force-vectorize=false.
+ if (!forceVectorize)
+ return;
+ // Hint the innermost loop around the RMW: for a nested reduction the match
+ // is at an outer loop but only the inner loop vectorizes.
+ fir::DoLoopOp hintLoop = rmw->getParentOfType<fir::DoLoopOp>();
+ // Bail if the body carries a loop that keeps the vectorizer out: one that
+ // reads the temporary (an inlined sum(a(:,i)) recurrence) or a non-
+ // unrollable inner loop. A small constant-trip scratch loop is fine.
+ bool blocked = false;
+ hintLoop->getRegion(0).walk([&](fir::DoLoopOp nested) {
+ bool usesTemp = llvm::any_of(temp.getUsers(), [&](mlir::Operation *user) {
+ return nested->isAncestor(user);
+ });
+ if (usesTemp || !isFullyUnrollableLoop(nested, maxUnrollableTripCount)) {
+ blocked = true;
+ return mlir::WalkResult::interrupt();
+ }
+ return mlir::WalkResult::advance();
+ });
+ if (blocked)
+ return;
+ mlir::MLIRContext *ctx = hintLoop.getContext();
+ mlir::LLVM::LoopAnnotationAttr existing = hintLoop.getLoopAnnotationAttr();
+ mlir::LLVM::LoopVectorizeAttr existingVec =
+ existing ? existing.getVectorize() : nullptr;
+
+ // Respect an explicit vectorize `disable = true` (e.g. from !DIR$
+ // NOVECTOR).
+ if (existingVec && existingVec.getDisable() &&
+ existingVec.getDisable().getValue())
+ return;
+
+ // Set only the vectorize-enable field, preserving any existing options.
+ mlir::BoolAttr enable = mlir::BoolAttr::get(ctx, /*disable=*/false);
+ mlir::LLVM::LoopVectorizeAttr vectorize =
+ existingVec
+ ? mlir::LLVM::LoopVectorizeAttr::get(
+ ctx, enable, existingVec.getPredicateEnable(),
+ existingVec.getScalableEnable(), existingVec.getWidth(),
+ existingVec.getFollowupVectorized(),
+ existingVec.getFollowupEpilogue(),
+ existingVec.getFollowupAll())
+ : mlir::LLVM::LoopVectorizeAttr::get(ctx, enable, {}, {}, {}, {},
+ {}, {});
+
+ // Merge into any existing annotation, preserving all non-vectorize fields
+ mlir::LLVM::LoopAnnotationAttr annotation =
+ existing ? mlir::LLVM::LoopAnnotationAttr::get(
+ ctx, existing.getDisableNonforced(), vectorize,
+ existing.getInterleave(), existing.getUnroll(),
+ existing.getUnrollAndJam(), existing.getLicm(),
+ existing.getDistribute(), existing.getPipeline(),
+ existing.getPeeled(), existing.getUnswitch(),
+ existing.getMustProgress(), existing.getIsVectorized(),
+ existing.getStartLoc(), existing.getEndLoc(),
+ existing.getParallelAccesses())
+ : mlir::LLVM::LoopAnnotationAttr::get(ctx, {}, vectorize, {},
+ {}, {}, {}, {}, {}, {},
+ {}, {}, {}, {}, {}, {});
+ hintLoop.setLoopAnnotationAttr(annotation);
+ }
+
+ /// Find the nearest unconditional hlfir.assign whose LHS designates the
+ /// identical whole section as \p rmwLhs and that dominates \p loop (P2). The
+ /// nearest (last dominating) define is required: with repeated same-section
+ /// reductions an earlier define could be a prior copy-out, not this loop's
+ /// init.
+ hlfir::AssignOp findDominatingFullDef(fir::DoLoopOp loop,
+ hlfir::DesignateOp rmwLhs,
+ mlir::DominanceInfo &domInfo) {
+ auto scanBlock = [&](mlir::Block *b) -> hlfir::AssignOp {
+ hlfir::AssignOp found;
+ for (mlir::Operation &op : *b) {
+ auto cand = mlir::dyn_cast<hlfir::AssignOp>(op);
+ if (!cand)
+ continue;
+ auto candLhs = cand.getLhs().getDefiningOp<hlfir::DesignateOp>();
+ if (!candLhs || !domInfo.dominates(cand, loop))
+ continue;
+ if (sameSection(candLhs, rmwLhs))
+ found = cand; // keep the last (nearest to the loop) match
+ }
+ return found;
+ };
+
+ // Only ops that dominate the loop can be the init, so restrict the search
+ // to the loop's block and its dominator-tree ancestors, walking out through
+ // the enclosing regions, instead of re-walking the whole function per
+ // candidate.
+ mlir::Operation *root = getOperation();
+ for (mlir::Operation *entry = loop.getOperation(); entry;) {
+ mlir::Block *block = entry->getBlock();
+ if (!block)
+ break;
+ mlir::Region *region = block->getParent();
+ // getNode asserts on single-block regions (its dom tree is not built);
+ // there the block is its own only dominator, so scan it directly.
+ if (region && !region->hasOneBlock()) {
+ for (mlir::DominanceInfoNode *node = domInfo.getNode(block); node;
+ node = node->getIDom())
+ if (hlfir::AssignOp init = scanBlock(node->getBlock()))
+ return init;
+ } else if (hlfir::AssignOp init = scanBlock(block)) {
+ return init;
+ }
+ if (entry == root || !region)
+ break;
+ entry = region->getParentOp();
+ }
+ return {};
+ }
+
+ /// P1: return true when the section's address is loop-invariant in \p loop,
+ /// i.e. no in-loop store or call may write a location its subscripts load
+ /// (e.g. the index i in a(:,i)). Not covered by P2, which matches addresses
+ /// structurally but not whether they are rewritten mid-loop.
+ bool sectionAddressLoopInvariant(fir::DoLoopOp loop,
+ llvm::ArrayRef<mlir::Value> addrDeps,
+ fir::AliasAnalysis &aliasAnalysis) {
+ if (addrDeps.empty())
+ return true; // address is a pure constant/SSA computation
+
+ // Whether op may write to any declared address dep. Specialized cases are
+ // handled directly; every other side-effecting op falls through to a
+ // generic memory-effects check so a write from e.g. fir.copy is not missed.
+ auto mayWriteDeps = [&](mlir::Operation *op) -> bool {
+ if (mlir::isMemoryEffectFree(op))
+ return false;
+ // hlfir.assign models a write to a box LHS as a Read of the descriptor,
+ // so getModRef under-reports it; check the resolved destination directly.
+ if (auto assign = mlir::dyn_cast<hlfir::AssignOp>(op)) {
+ for (mlir::Value dep : addrDeps)
+ if (!aliasAnalysis.alias(assign.getLhs(), dep).isNo())
+ return true;
+ return false;
+ }
+ // Region carriers (including fir.do_loop, which is opaque to getModRef)
+ // are traversed by the walk; apply the canonical policy to leaf ops.
+ // getModRef fails closed for an opaque leaf and unresolved writes.
+ if (op->getNumRegions() != 0)
+ return false;
+ for (mlir::Value dep : addrDeps)
+ if (aliasAnalysis.getModRef(op, dep).isMod())
+ return true;
+ return false;
+ };
+
+ mlir::WalkResult result = loop.walk([&](mlir::Operation *op) {
+ return mayWriteDeps(op) ? mlir::WalkResult::interrupt()
+ : mlir::WalkResult::advance();
+ });
+ return !result.wasInterrupted();
+ }
+
+ /// P2 (interval): return true when nothing between the dominating init and
+ /// the loop accesses the section. promote() folds the init away and
+ /// overwrites the section with a post-loop copy-out, so a read or write there
+ /// -- or a write to a subscript the address depends on -- would be
+ /// miscompiled.
+ bool sectionUntouchedBetweenInitAndLoop(hlfir::AssignOp init,
+ fir::DoLoopOp loop, hlfir::Entity sec,
+ llvm::ArrayRef<mlir::Value> addrDeps,
+ mlir::DominanceInfo &domInfo,
+ fir::AliasAnalysis &aliasAnalysis) {
+ mlir::Operation *initOp = init.getOperation();
+ mlir::Operation *loopOp = loop.getOperation();
+
+ // Whether op itself accesses the section or writes one of its subscripts.
+ auto touchesOp = [&](mlir::Operation *op) -> bool {
+ if (mlir::isMemoryEffectFree(op))
+ return false;
+ // hlfir.assign models a write to a box LHS as a Read of the descriptor,
+ // so getModRef under-reports it; check the resolved destination directly.
+ if (auto asg = mlir::dyn_cast<hlfir::AssignOp>(op)) {
+ if (!aliasAnalysis.alias(sec, asg.getLhs()).isNo())
+ return true;
+ // A direct copy `other = section` reads the section's original storage
+ // through the RHS variable; promote() folds the init away and leaves
+ // that storage stale. (An expr RHS reads memory only through inner ops,
+ // which the walk visits, so restrict this to a variable RHS.)
+ mlir::Value rhs = asg.getRhs();
+ if (hlfir::Entity(rhs).isVariable() &&
+ !aliasAnalysis.alias(sec, rhs).isNo())
+ return true;
+ for (mlir::Value dep : addrDeps)
+ if (!aliasAnalysis.alias(asg.getLhs(), dep).isNo())
+ return true;
+ return false;
+ }
+ // Region carriers (including fir.do_loop, which is opaque to getModRef)
+ // are traversed by touchesTree; a leaf touches the section if it reads or
+ // writes it, or writes a subscript. getModRef fails closed for an opaque
+ // leaf and unresolved effects.
+ if (op->getNumRegions() != 0)
+ return false;
+ if (aliasAnalysis.getModRef(op, sec).isModOrRef())
+ return true;
+ for (mlir::Value dep : addrDeps)
+ if (aliasAnalysis.getModRef(op, dep).isMod())
+ return true;
+ return false;
+ };
+
+ auto touchesTree = [&](mlir::Operation *root) {
+ bool found = false;
+ root->walk([&](mlir::Operation *op) {
+ if (!found && touchesOp(op))
+ found = true;
+ });
+ return found;
+ };
+
+ // Common case: the init and the loop are sequential in one block (the init
+ // dominates the loop, so it precedes it). The interval is then exactly the
+ // ops between them -- scan them linearly, skipping the whole-scope walk and
+ // per-op dominance queries of the general path below.
+ if (initOp->getBlock() == loopOp->getBlock()) {
+ for (mlir::Operation *op = initOp->getNextNode(); op && op != loopOp;
+ op = op->getNextNode())
+ if (touchesTree(op))
+ return false;
+ return true;
+ }
+
+ mlir::Operation *scope = initOp->getParentOp();
+ if (!scope)
+ return true;
+ bool clear = true;
+ scope->walk([&](mlir::Operation *op) {
+ if (!clear || op == initOp || op == loopOp)
+ return;
+ // "Between" = dominated by the init and not strictly after the loop, so
+ // conditionally executed accesses in the init->loop window are included.
+ if (domInfo.properlyDominates(initOp, op) &&
+ !domInfo.properlyDominates(loopOp, op) && touchesTree(op))
+ clear = false;
+ });
+ return clear;
+ }
+
+ /// P3 (writes): return true when nothing inside \p loop other than \p rmw may
+ /// clobber the promoted section \p sec. Any surviving in-loop write to the
+ /// section -- an hlfir.assign, a direct fir.array_coor+fir.store, or a call
+ /// -- would be silently overwritten by the post-loop copy-out of T, so it
+ /// blocks promotion. Writes are found via memory effects, failing closed for
+ /// an unresolved write or an opaque side-effecting leaf.
+ bool noAliasingWritesInLoop(fir::DoLoopOp loop, hlfir::AssignOp rmw,
+ hlfir::Entity sec,
+ fir::AliasAnalysis &aliasAnalysis) {
+ mlir::WalkResult result = loop.walk([&](mlir::Operation *op) {
+ if (op == rmw.getOperation() || mlir::isMemoryEffectFree(op))
+ return mlir::WalkResult::advance();
+
+ // hlfir.assign models a write to a box LHS as a Read of the descriptor,
+ // so getModRef under-reports it; check the resolved destination directly.
+ if (auto other = mlir::dyn_cast<hlfir::AssignOp>(op)) {
+ if (!aliasAnalysis.alias(sec, other.getLhs()).isNo())
+ return mlir::WalkResult::interrupt();
+ return mlir::WalkResult::advance();
+ }
+ // A call's read of the section cannot be redirected to the temporary, so
+ // reject a call that modifies OR references it (not just writes).
+ if (mlir::isa<fir::CallOp, fir::DispatchOp>(op)) {
+ if (aliasAnalysis.getModRef(op, sec).isModOrRef())
+ return mlir::WalkResult::interrupt();
+ return mlir::WalkResult::advance();
+ }
+
+ // Region carriers (including fir.do_loop, which is opaque to getModRef)
+ // are traversed by the walk; a leaf that may write the section clobbers
+ // it. This catches direct writes such as fir.array_coor + fir.store, and
+ // getModRef fails closed for an opaque leaf and unresolved writes.
+ if (op->getNumRegions() != 0)
+ return mlir::WalkResult::advance();
+ if (aliasAnalysis.getModRef(op, sec).isMod())
+ return mlir::WalkResult::interrupt();
+ return mlir::WalkResult::advance();
+ });
+ return !result.wasInterrupted();
+ }
+
+ /// P3 (reads): return true when every in-loop read that may alias the
+ /// promoted section \p sec flows through a designator the rewrite redirects
+ /// to the temporary (the RMW LHS \p lhsDes or a collected read, or one built
+ /// on them). Any other aliasing read -- an element read a(k,i), an
+ /// overlapping slice, or a direct fir.load / fir.copy / fir.array_load of the
+ /// section -- would read stale memory once promoted.
+ bool allAliasingAccessesRedirected(fir::DoLoopOp loop, hlfir::Entity sec,
+ hlfir::DesignateOp lhsDes,
+ llvm::ArrayRef<hlfir::DesignateOp> reads,
+ fir::AliasAnalysis &aliasAnalysis) {
+ llvm::SmallPtrSet<mlir::Operation *, 8> redirected;
+ redirected.insert(lhsDes.getOperation());
+ for (hlfir::DesignateOp d : reads)
+ redirected.insert(d.getOperation());
+
+ // A designator is safe if it or a designator it is built on is redirected;
+ // replaceUsesWithIf rewires the whole chain to the temporary.
+ auto derivesFromRedirected = [&](hlfir::DesignateOp d) {
+ mlir::Value base = d.getResult();
+ while (auto des = base.getDefiningOp<hlfir::DesignateOp>()) {
+ if (redirected.contains(des.getOperation()))
+ return true;
+ base = des.getMemref();
+ }
+ return false;
+ };
+ // A read of \p v is safe when v flows through a redirected designator.
+ auto readsRedirected = [&](mlir::Value v) {
+ auto d = v.getDefiningOp<hlfir::DesignateOp>();
+ return d && derivesFromRedirected(d);
+ };
+
+ mlir::WalkResult result = loop.walk([&](mlir::Operation *op) {
+ // A section-aliasing designator must be one the rewrite redirects to T.
+ if (auto d = mlir::dyn_cast<hlfir::DesignateOp>(op)) {
+ if (!derivesFromRedirected(d) &&
+ !aliasAnalysis.alias(sec, d.getResult()).isNo())
+ return mlir::WalkResult::interrupt();
+ return mlir::WalkResult::advance();
+ }
+ if (mlir::isMemoryEffectFree(op))
+ return mlir::WalkResult::advance();
+ // Any other op: a read that may alias the section must flow through a
+ // redirected designator (fir.load, fir.copy, fir.array_load, ...) or it
+ // would observe stale memory once the reduction accumulates into T.
+ if (auto iface = mlir::dyn_cast<mlir::MemoryEffectOpInterface>(op)) {
+ llvm::SmallVector<mlir::MemoryEffects::EffectInstance, 4> effects;
+ iface.getEffects(effects);
+ for (const mlir::MemoryEffects::EffectInstance &e : effects) {
+ if (!mlir::isa<mlir::MemoryEffects::Read>(e.getEffect()))
+ continue;
+ mlir::Value v = e.getValue();
+ if (v && readsRedirected(v))
+ continue;
+ // An unresolved read (no value) could read anything: fail closed.
+ if (!v || !aliasAnalysis.alias(sec, v).isNo())
+ return mlir::WalkResult::interrupt();
+ }
+ return mlir::WalkResult::advance();
+ }
+ // No memory-effect interface: safe only if it is a region carrier whose
+ // body is walked separately; an opaque leaf might read anything.
+ if (op->getNumRegions() == 0)
+ return mlir::WalkResult::interrupt();
+ return mlir::WalkResult::advance();
+ });
+ return !result.wasInterrupted();
+ }
+
+ /// P4: total element count of the RHS when it has a compile-time-constant
+ /// shape, or nullopt for a scalar or dynamic-extent RHS.
+ static std::optional<int64_t> rhsConstantExtent(mlir::Value rhs) {
+ auto exprTy = mlir::dyn_cast<hlfir::ExprType>(rhs.getType());
+ if (!exprTy || exprTy.isScalar())
+ return std::nullopt;
+ int64_t count = 1;
+ for (int64_t e : exprTy.getShape()) {
+ // Overflow means an enormous shape, which the extent cap rejects anyway.
+ if (mlir::ShapedType::isDynamic(e) || llvm::MulOverflow(count, e, count))
+ return std::nullopt;
+ }
+ return count;
+ }
+};
+
+} // namespace
diff --git a/flang/lib/Optimizer/HLFIR/Transforms/CMakeLists.txt b/flang/lib/Optimizer/HLFIR/Transforms/CMakeLists.txt
index 15b4593166d0c..1e1331bed7ba6 100644
--- a/flang/lib/Optimizer/HLFIR/Transforms/CMakeLists.txt
+++ b/flang/lib/Optimizer/HLFIR/Transforms/CMakeLists.txt
@@ -1,6 +1,7 @@
get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
add_flang_library(HLFIRTransforms
+ ArraySectionReduction.cpp
BufferizeHLFIR.cpp
ConvertToFIR.cpp
ExpressionSimplification.cpp
diff --git a/flang/lib/Optimizer/Passes/Pipelines.cpp b/flang/lib/Optimizer/Passes/Pipelines.cpp
index 15a342e10fc7f..257de61c65c74 100644
--- a/flang/lib/Optimizer/Passes/Pipelines.cpp
+++ b/flang/lib/Optimizer/Passes/Pipelines.cpp
@@ -290,6 +290,17 @@ void createHLFIRToFIRPassPipeline(mlir::PassManager &pm,
});
addNestedPassToAllTopLevelOperations<PassConstructor>(
pm, hlfir::createPropagateFortranVariableAttributes);
+ // Run createArraySectionReduction pass before OptimizedBufferization/
+ // InlineHLFIRAssign, while the reduction is still a single hlfir.assign
+ // of a designate. Opt-in via -fexperimental-array-section-reduction. Only
+ // request the vectorization hint at O2/O3, where the LLVM loop vectorizer
+ // runs; the promotion itself still applies at O1.
+ if (config.ArraySectionReduction)
+ addNestedPassToAllTopLevelOperations(pm, [&]() {
+ return hlfir::createArraySectionReduction(
+ {/*forceVectorize=*/optLevel == llvm::OptimizationLevel::O2 ||
+ optLevel == llvm::OptimizationLevel::O3});
+ });
addNestedPassToAllTopLevelOperations<PassConstructor>(
pm, hlfir::createOptimizedBufferization);
addNestedPassToAllTopLevelOperations<PassConstructor>(
diff --git a/flang/test/Driver/array-section-reduction.f90 b/flang/test/Driver/array-section-reduction.f90
new file mode 100644
index 0000000000000..f43b85d399d6e
--- /dev/null
+++ b/flang/test/Driver/array-section-reduction.f90
@@ -0,0 +1,46 @@
+! Test driver handling of -fexperimental-array-section-reduction and
+! -fno-experimental-array-section-reduction.
+
+! RUN: %flang -fsyntax-only -### %s -o %t 2>&1 \
+! RUN: | FileCheck %s --check-prefix=DISABLED
+
+! RUN: %flang -fsyntax-only -### %s -o %t 2>&1 \
+! RUN: -fexperimental-array-section-reduction \
+! RUN: | FileCheck %s --check-prefix=ENABLED
+
+! RUN: %flang -fsyntax-only -### %s -o %t 2>&1 \
+! RUN: -fno-experimental-array-section-reduction \
+! RUN: | FileCheck %s --check-prefix=DISABLED
+
+! RUN: %flang -fsyntax-only -### %s -o %t 2>&1 \
+! RUN: -fno-experimental-array-section-reduction -fexperimental-array-section-reduction \
+! RUN: | FileCheck %s --check-prefix=ENABLED
+
+! RUN: %flang -fsyntax-only -### %s -o %t 2>&1 \
+! RUN: -fexperimental-array-section-reduction -fno-experimental-array-section-reduction \
+! RUN: | FileCheck %s --check-prefix=DISABLED
+
+! DISABLED: "-fc1"
+! DISABLED-NOT: "-fexperimental-array-section-reduction"
+
+! ENABLED: "-fc1"
+! ENABLED-SAME: "-fexperimental-array-section-reduction"
+
+! Prove the flag actually inserts the pass into the pipeline. This is an MLIR
+! HLFIR pass, so it is invisible to LLVM's -mllvm -print-pipeline-passes (that
+! prints the LLVM pass pipeline, which is unchanged by an MLIR pass); dump the
+! scheduled MLIR pass pipeline instead -- the same mechanism mlir-pass-pipeline.f90
+! uses -- and check that ArraySectionReduction is listed only with the flag. The
+! pass is scheduled at O1+, so -O2 is used. Requires asserts for --mlir-pass-statistics.
+
+! RUN: %if asserts %{ %flang_fc1 -S -O2 -fexperimental-array-section-reduction \
+! RUN: -mmlir --mlir-pass-statistics -mmlir --mlir-pass-statistics-display=pipeline \
+! RUN: %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=INSERTED %}
+! RUN: %if asserts %{ %flang_fc1 -S -O2 \
+! RUN: -mmlir --mlir-pass-statistics -mmlir --mlir-pass-statistics-display=pipeline \
+! RUN: %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=ABSENT %}
+
+! INSERTED: ArraySectionReduction
+! ABSENT-NOT: ArraySectionReduction
+
+end program
diff --git a/flang/test/HLFIR/array-section-reduction-extent.fir b/flang/test/HLFIR/array-section-reduction-extent.fir
new file mode 100644
index 0000000000000..c8e49de30a001
--- /dev/null
+++ b/flang/test/HLFIR/array-section-reduction-extent.fir
@@ -0,0 +1,86 @@
+// Test the max-reduction-extent pass option, which bounds the constant section
+// extent that promotion will accept. A section whose extent exceeds the option
+// is detected but left un-promoted (no constant-shape temporary is created).
+// RUN: fir-opt --split-input-file --array-section-reduction %s \
+// RUN: | FileCheck %s --check-prefix=DEFAULT
+// RUN: fir-opt --split-input-file --array-section-reduction=max-reduction-extent=9 %s \
+// RUN: | FileCheck %s --check-prefix=EXT9
+
+// Extent 8 sits exactly on the default boundary (8): promoted.
+// DEFAULT-LABEL: func.func @_QPext8
+// DEFAULT: fir.alloca !fir.array<8xf64>
+func.func @_QPext8(%arg0: !fir.ref<!fir.array<8xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<8xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c8 = arith.constant 8 : index
+ %sh = fir.shape %c8 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh) dummy_scope %0 arg 1 {uniq_name = "_QFext8Ea"} : (!fir.ref<!fir.array<8xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<8xf64>>, !fir.ref<!fir.array<8xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFext8Exx"} : (!fir.ref<!fir.array<8xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<8xf64>>, !fir.ref<!fir.array<8xf64>>)
+ %n:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFext8En"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %cst = arith.constant 0.000000e+00 : f64
+ %c1 = arith.constant 1 : index
+ %init = hlfir.designate %a#0 (%c1:%c8:%c1) shape %sh : (!fir.ref<!fir.array<8xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<8xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<8xf64>>
+ %c1_i32 = arith.constant 1 : i32
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %rd = hlfir.designate %a#0 (%c1:%c8:%c1) shape %sh : (!fir.ref<!fir.array<8xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<8xf64>>
+ %xxs = hlfir.designate %xx#0 (%c1:%c8:%c1) shape %sh : (!fir.ref<!fir.array<8xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<8xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<8xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<8xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xxs (%k) : (!fir.ref<!fir.array<8xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c8:%c1) shape %sh : (!fir.ref<!fir.array<8xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<8xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<8xf64>, !fir.ref<!fir.array<8xf64>>
+ hlfir.destroy %el : !hlfir.expr<8xf64>
+ }
+ return
+}
+
+// -----
+
+// Extent 9 is one past the default boundary: rejected at the default (8), and
+// promoted only when max-reduction-extent is raised to 9.
+// DEFAULT-LABEL: func.func @_QPext9
+// DEFAULT-NOT: fir.alloca !fir.array
+// EXT9-LABEL: func.func @_QPext9
+// EXT9: fir.alloca !fir.array<9xf64>
+func.func @_QPext9(%arg0: !fir.ref<!fir.array<9xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<9xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c9 = arith.constant 9 : index
+ %sh = fir.shape %c9 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh) dummy_scope %0 arg 1 {uniq_name = "_QFext9Ea"} : (!fir.ref<!fir.array<9xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<9xf64>>, !fir.ref<!fir.array<9xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFext9Exx"} : (!fir.ref<!fir.array<9xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<9xf64>>, !fir.ref<!fir.array<9xf64>>)
+ %n:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFext9En"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %cst = arith.constant 0.000000e+00 : f64
+ %c1 = arith.constant 1 : index
+ %init = hlfir.designate %a#0 (%c1:%c9:%c1) shape %sh : (!fir.ref<!fir.array<9xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<9xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<9xf64>>
+ %c1_i32 = arith.constant 1 : i32
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %rd = hlfir.designate %a#0 (%c1:%c9:%c1) shape %sh : (!fir.ref<!fir.array<9xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<9xf64>>
+ %xxs = hlfir.designate %xx#0 (%c1:%c9:%c1) shape %sh : (!fir.ref<!fir.array<9xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<9xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<9xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<9xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xxs (%k) : (!fir.ref<!fir.array<9xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c9:%c1) shape %sh : (!fir.ref<!fir.array<9xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<9xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<9xf64>, !fir.ref<!fir.array<9xf64>>
+ hlfir.destroy %el : !hlfir.expr<9xf64>
+ }
+ return
+}
diff --git a/flang/test/HLFIR/array-section-reduction.fir b/flang/test/HLFIR/array-section-reduction.fir
new file mode 100644
index 0000000000000..e2e895ae13b14
--- /dev/null
+++ b/flang/test/HLFIR/array-section-reduction.fir
@@ -0,0 +1,1896 @@
+// Test the loop-invariant array-section reduction promotion pass. It rewrites a
+// whole-section reduction (a(:,i) = 0; do j; a(:,i) = a(:,i) + ...) to
+// accumulate into a constant-shape local temporary and, by default, annotates
+// the loop to enable vectorization. Positive cases check the rewrite; negative
+// cases check that each precondition blocks promotion.
+// RUN: fir-opt --split-input-file --array-section-reduction %s | FileCheck %s
+// RUN: fir-opt --split-input-file --array-section-reduction=force-vectorize=false %s | FileCheck %s --check-prefix=NOVEC
+
+// Fortran source of the canonical promotable reduction (@_QPpos):
+// subroutine pos(a, x, xx, i, n)
+// real(8) :: a(:,:), x(:), xx(3)
+// integer :: i, n, j
+// a(:,i) = 0.0
+// do j = 1, n
+// a(:,i) = a(:,i) + xx*x(j) ! reduction into a loop-invariant section
+// end do
+// end subroutine
+// Each case below is this pattern with one change: positives must still promote,
+// negatives must be blocked by the noted precondition.
+
+// @_QPpos: all preconditions hold -> promoted.
+// The pass hoists a constant-shape temporary %[[T]], folds the init into it,
+// redirects the in-loop read-modify-write to it, copies it back after the loop,
+// and annotates the loop for vectorization.
+// CHECK: #[[VEC:.*]] = #llvm.loop_vectorize<disable = false>
+// CHECK: #[[ANNOT:.*]] = #llvm.loop_annotation<vectorize = #[[VEC]]>
+// CHECK-LABEL: func.func @_QPpos
+// CHECK: %[[T:.*]] = fir.alloca !fir.array<3xf64>
+// CHECK: hlfir.assign %{{.*}} to %[[T]] : f64, !fir.ref<!fir.array<3xf64>>
+// CHECK: fir.do_loop {{.*}}attributes {loopAnnotation = #[[ANNOT]]}
+// CHECK: hlfir.designate %[[T]] (%{{.*}}) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+// CHECK: hlfir.assign %{{.*}} to %[[T]] : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+// CHECK: hlfir.assign %[[T]] to %{{.*}} : !fir.ref<!fir.array<3xf64>>, !fir.box<!fir.array<?xf64>>
+// With -force-vectorize=false the section is still promoted to the temporary,
+// but the loop is not annotated: the vectorization hint is separable.
+// NOVEC-LABEL: func.func @_QPpos
+// NOVEC: %[[T:.*]] = fir.alloca !fir.array<3xf64>
+// NOVEC: hlfir.assign %{{.*}} to %[[T]] : f64, !fir.ref<!fir.array<3xf64>>
+// NOVEC: hlfir.assign %[[T]] to %{{.*}} : !fir.ref<!fir.array<3xf64>>, !fir.box<!fir.array<?xf64>>
+// NOVEC-NOT: loopAnnotation
+func.func @_QPpos(%arg0: !fir.box<!fir.array<?x?xf64>> {fir.bindc_name = "a"}, %arg1: !fir.box<!fir.array<?xf64>> {fir.bindc_name = "x"}, %arg2: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg3: !fir.ref<i32> {fir.bindc_name = "i"}, %arg4: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %1:2 = hlfir.declare %arg0 dummy_scope %0 arg 1 {uniq_name = "_QFposEa"} : (!fir.box<!fir.array<?x?xf64>>, !fir.dscope) -> (!fir.box<!fir.array<?x?xf64>>, !fir.box<!fir.array<?x?xf64>>)
+ %2:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFposEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %3 = fir.alloca i32 {bindc_name = "j", uniq_name = "_QFposEj"}
+ %4:2 = hlfir.declare %3 {uniq_name = "_QFposEj"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %5:2 = hlfir.declare %arg4 dummy_scope %0 arg 5 {uniq_name = "_QFposEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %6:2 = hlfir.declare %arg1 dummy_scope %0 arg 2 {uniq_name = "_QFposEx"} : (!fir.box<!fir.array<?xf64>>, !fir.dscope) -> (!fir.box<!fir.array<?xf64>>, !fir.box<!fir.array<?xf64>>)
+ %c3 = arith.constant 3 : index
+ %7 = fir.shape %c3 : (index) -> !fir.shape<1>
+ %8:2 = hlfir.declare %arg2(%7) dummy_scope %0 arg 3 {uniq_name = "_QFposExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %cst = arith.constant 0.000000e+00 : f64
+ %c1 = arith.constant 1 : index
+ %c0 = arith.constant 0 : index
+ %9:3 = fir.box_dims %1#1, %c0 : (!fir.box<!fir.array<?x?xf64>>, index) -> (index, index, index)
+ %c1_0 = arith.constant 1 : index
+ %c0_1 = arith.constant 0 : index
+ %10 = arith.subi %9#1, %c1 : index
+ %11 = arith.addi %10, %c1_0 : index
+ %12 = arith.divsi %11, %c1_0 : index
+ %13 = arith.cmpi sgt, %12, %c0_1 : index
+ %14 = arith.select %13, %12, %c0_1 : index
+ %15 = fir.load %2#0 : !fir.ref<i32>
+ %16 = fir.convert %15 : (i32) -> i64
+ %17 = fir.shape %14 : (index) -> !fir.shape<1>
+ %18 = hlfir.designate %1#0 (%c1:%9#1:%c1_0, %16) shape %17 : (!fir.box<!fir.array<?x?xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.box<!fir.array<?xf64>>
+ hlfir.assign %cst to %18 : f64, !fir.box<!fir.array<?xf64>>
+ %c1_i32 = arith.constant 1 : i32
+ %19 = fir.convert %c1_i32 : (i32) -> index
+ %20 = fir.load %5#0 : !fir.ref<i32>
+ %21 = fir.convert %20 : (i32) -> index
+ %c1_2 = arith.constant 1 : index
+ %22 = fir.convert %19 : (index) -> i32
+ %23 = fir.do_loop %arg5 = %19 to %21 step %c1_2 iter_args(%arg6 = %22) -> (i32) {
+ fir.store %arg6 to %4#0 : !fir.ref<i32>
+ %c1_3 = arith.constant 1 : index
+ %c0_4 = arith.constant 0 : index
+ %24:3 = fir.box_dims %1#1, %c0_4 : (!fir.box<!fir.array<?x?xf64>>, index) -> (index, index, index)
+ %c1_5 = arith.constant 1 : index
+ %c0_6 = arith.constant 0 : index
+ %25 = arith.subi %24#1, %c1_3 : index
+ %26 = arith.addi %25, %c1_5 : index
+ %27 = arith.divsi %26, %c1_5 : index
+ %28 = arith.cmpi sgt, %27, %c0_6 : index
+ %29 = arith.select %28, %27, %c0_6 : index
+ %30 = fir.load %2#0 : !fir.ref<i32>
+ %31 = fir.convert %30 : (i32) -> i64
+ %32 = fir.shape %29 : (index) -> !fir.shape<1>
+ %33 = hlfir.designate %1#0 (%c1_3:%24#1:%c1_5, %31) shape %32 : (!fir.box<!fir.array<?x?xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.box<!fir.array<?xf64>>
+ %34 = fir.load %4#0 : !fir.ref<i32>
+ %35 = fir.convert %34 : (i32) -> i64
+ %36 = hlfir.designate %6#0 (%35) : (!fir.box<!fir.array<?xf64>>, i64) -> !fir.ref<f64>
+ %37 = fir.load %36 : !fir.ref<f64>
+ %c1_7 = arith.constant 1 : index
+ %c1_8 = arith.constant 1 : index
+ %c3_9 = arith.constant 3 : index
+ %38 = fir.shape %c3_9 : (index) -> !fir.shape<1>
+ %39 = hlfir.designate %8#0 (%c1_7:%c3:%c1_8) shape %38 : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %40 = hlfir.elemental %38 unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%arg7: index):
+ %55 = hlfir.designate %39 (%arg7) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %56 = fir.load %55 : !fir.ref<f64>
+ %57 = arith.mulf %37, %56 fastmath<contract> : f64
+ hlfir.yield_element %57 : f64
+ }
+ %41 = hlfir.elemental %38 unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%arg7: index):
+ %55 = hlfir.designate %33 (%arg7) : (!fir.box<!fir.array<?xf64>>, index) -> !fir.ref<f64>
+ %56 = hlfir.apply %40, %arg7 : (!hlfir.expr<3xf64>, index) -> f64
+ %57 = fir.load %55 : !fir.ref<f64>
+ %58 = arith.addf %57, %56 fastmath<contract> : f64
+ hlfir.yield_element %58 : f64
+ }
+ %c1_10 = arith.constant 1 : index
+ %c0_11 = arith.constant 0 : index
+ %42:3 = fir.box_dims %1#1, %c0_11 : (!fir.box<!fir.array<?x?xf64>>, index) -> (index, index, index)
+ %c1_12 = arith.constant 1 : index
+ %c0_13 = arith.constant 0 : index
+ %43 = arith.subi %42#1, %c1_10 : index
+ %44 = arith.addi %43, %c1_12 : index
+ %45 = arith.divsi %44, %c1_12 : index
+ %46 = arith.cmpi sgt, %45, %c0_13 : index
+ %47 = arith.select %46, %45, %c0_13 : index
+ %48 = fir.load %2#0 : !fir.ref<i32>
+ %49 = fir.convert %48 : (i32) -> i64
+ %50 = fir.shape %47 : (index) -> !fir.shape<1>
+ %51 = hlfir.designate %1#0 (%c1_10:%42#1:%c1_12, %49) shape %50 : (!fir.box<!fir.array<?x?xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.box<!fir.array<?xf64>>
+ hlfir.assign %41 to %51 : !hlfir.expr<3xf64>, !fir.box<!fir.array<?xf64>>
+ hlfir.destroy %41 : !hlfir.expr<3xf64>
+ hlfir.destroy %40 : !hlfir.expr<3xf64>
+ %52 = fir.convert %c1_2 : (index) -> i32
+ %53 = fir.load %4#0 : !fir.ref<i32>
+ %54 = arith.addi %53, %52 overflow<nsw> : i32
+ fir.result %54 : i32
+ }
+ fir.store %23 to %4#0 : !fir.ref<i32>
+ return
+}
+// -----
+
+// @_QPpos_mixed: mixed-precision RMW, minimized to the pattern that triggers
+// promotion. a is real(4) but the reduction expression is real(8), so the RHS
+// is !hlfir.expr<3xf64> while the section is f32. The promoted temporary must
+// take the section element type (f32), not the RHS's (f64), or the redirected
+// reads would be mis-typed. Source: real(4) a(3); real(8) xx(3);
+// a = 0.0; do j = 1, n; a = a + xx; end do (a promotes to real(8)).
+// CHECK-LABEL: func.func @_QPpos_mixed
+// CHECK: %[[T:.*]] = fir.alloca !fir.array<3xf32>
+// CHECK: hlfir.assign %{{.*}} to %[[T]] : f32, !fir.ref<!fir.array<3xf32>>
+// CHECK: hlfir.designate %[[T]] (%{{.*}}) : (!fir.ref<!fir.array<3xf32>>, index) -> !fir.ref<f32>
+// CHECK: hlfir.assign %{{.*}} to %[[T]] : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf32>>
+// CHECK: hlfir.assign %[[T]] to %{{.*}} : !fir.ref<!fir.array<3xf32>>, !fir.ref<!fir.array<3xf32>>
+func.func @_QPpos_mixed(%arg0: !fir.ref<!fir.array<3xf32>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c3 = arith.constant 3 : index
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh) dummy_scope %0 arg 1 {uniq_name = "_QFpos_mixedEa"} : (!fir.ref<!fir.array<3xf32>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf32>>, !fir.ref<!fir.array<3xf32>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFpos_mixedExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %n:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFpos_mixedEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %cst = arith.constant 0.000000e+00 : f32
+ %c1 = arith.constant 1 : index
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf32>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf32>>
+ hlfir.assign %cst to %init : f32, !fir.ref<!fir.array<3xf32>>
+ %c1_i32 = arith.constant 1 : i32
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf32>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf32>>
+ %xxs = hlfir.designate %xx#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf32>>, index) -> !fir.ref<f32>
+ %av = fir.load %ae : !fir.ref<f32>
+ %acv = fir.convert %av : (f32) -> f64
+ %xe = hlfir.designate %xxs (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %acv, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf32>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf32>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf32>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ // Unrelated dynamic heap free: its !fir.heap value must reach alias() raw
+ // (not wrapped in hlfir::Entity, which would assert on the non-entity type).
+ %h = fir.allocmem !fir.array<?xf64>, %ub
+ fir.freemem %h : !fir.heap<!fir.array<?xf64>>
+ }
+ return
+}
+// -----
+
+// a(2:3,i) = a(2:3,i) + ... (partial slice, not the whole section a(:,i))
+// P2: no dominating full-section define matches -> not promoted.
+// CHECK-LABEL: func.func @_QPneg_partial
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_partial(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c2 = arith.constant 2 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %shp = fir.shape %c2 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFneg_partialEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFneg_partialExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %i:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFneg_partialEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFneg_partialEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %iv = fir.load %i#0 : !fir.ref<i32>
+ %ic = fir.convert %iv : (i32) -> i64
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ // Partial slice a(2:3,i), not the whole section a(:,i) the init defines.
+ %rd = hlfir.designate %a#0 (%c2:%c3:%c1, %icl) shape %shp : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<2xf64>>
+ %xxs = hlfir.designate %xx#0 (%c2:%c3:%c1) shape %shp : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<2xf64>>
+ %el = hlfir.elemental %shp unordered : (!fir.shape<1>) -> !hlfir.expr<2xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<2xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xxs (%k) : (!fir.ref<!fir.array<2xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c2:%c3:%c1, %icl) shape %shp : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<2xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<2xf64>, !fir.ref<!fir.array<2xf64>>
+ hlfir.destroy %el : !hlfir.expr<2xf64>
+ }
+ return
+}
+// -----
+
+// No a(:,i) = 0 before the loop.
+// P2: no dominating full-section define -> not promoted.
+// CHECK-LABEL: func.func @_QPneg_noinit
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_noinit(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFneg_noinitEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFneg_noinitExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %i:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFneg_noinitEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFneg_noinitEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ // No a(:,i) = 0 init before the loop.
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xx#0 (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+// -----
+
+// x(j) = a(1,i) an extra element read of the section, outside the RMW.
+// P3 (reads): that read would see stale memory once promoted -> not promoted.
+// CHECK-LABEL: func.func @_QPneg_extra_read
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_extra_read(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}, %arg4: !fir.ref<f64> {fir.bindc_name = "res"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFneg_extra_readEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFneg_extra_readExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %i:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFneg_extra_readEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFneg_extra_readEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %res:2 = hlfir.declare %arg4 dummy_scope %0 arg 5 {uniq_name = "_QFneg_extra_readEres"} : (!fir.ref<f64>, !fir.dscope) -> (!fir.ref<f64>, !fir.ref<f64>)
+ %iv = fir.load %i#0 : !fir.ref<i32>
+ %ic = fir.convert %iv : (i32) -> i64
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ // An extra element read a(1,i) outside the RMW: P3-reads rejects, since it
+ // would observe stale memory once the reduction accumulates into T.
+ %ivl0 = fir.load %i#0 : !fir.ref<i32>
+ %icl0 = fir.convert %ivl0 : (i32) -> i64
+ %er = hlfir.designate %a#0 (%c1, %icl0) : (!fir.ref<!fir.array<3x4xf64>>, index, i64) -> !fir.ref<f64>
+ %erv = fir.load %er : !fir.ref<f64>
+ hlfir.assign %erv to %res#0 : f64, !fir.ref<f64>
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xx#0 (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+// -----
+
+// call sink(a) inside the loop, may modify or read the section.
+// P3 (writes): a call touching the section bypasses the temporary -> not promoted.
+// @_QPsink lives in this chunk so the split-out module resolves it.
+// CHECK-LABEL: func.func @_QPneg_call
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func private @_QPsink(!fir.ref<!fir.array<3x4xf64>>)
+func.func @_QPneg_call(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFneg_callEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFneg_callExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %i:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFneg_callEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFneg_callEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %iv = fir.load %i#0 : !fir.ref<i32>
+ %ic = fir.convert %iv : (i32) -> i64
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ // A call that may modify or read the section bypasses the temporary.
+ fir.call @_QPsink(%a#0) : (!fir.ref<!fir.array<3x4xf64>>) -> ()
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xx#0 (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+// -----
+
+// i = i + 1 inside the loop: the section address is not loop-invariant.
+// P1: a subscript the section reads is rewritten mid-loop -> not promoted.
+// CHECK-LABEL: func.func @_QPneg_index
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_index(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFneg_indexEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFneg_indexExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %i:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFneg_indexEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFneg_indexEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %iv = fir.load %i#0 : !fir.ref<i32>
+ %ic = fir.convert %iv : (i32) -> i64
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ // Reassign the section index i mid-loop, so the section address is not
+ // loop-invariant and P1 rejects.
+ %oi = fir.load %i#0 : !fir.ref<i32>
+ %ni = arith.addi %oi, %c1_i32 : i32
+ fir.store %ni to %i#0 : !fir.ref<i32>
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xx#0 (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+
+// -----
+
+// if (j > 0) a(:,i) = a(:,i) + x(j) (conditional RMW; init still unconditional)
+// Promotes: the dominating init keeps promotion sound even when the RMW is guarded.
+// CHECK: #[[VEC:.*]] = #llvm.loop_vectorize<disable = false>
+// CHECK: #[[ANNOT:.*]] = #llvm.loop_annotation<vectorize = #[[VEC]]>
+// CHECK-LABEL: func.func @_QPpos_cond
+// CHECK: fir.alloca !fir.array<3xf64>
+// CHECK: fir.do_loop {{.*}}attributes {loopAnnotation = #[[ANNOT]]}
+func.func @_QPpos_cond(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c0 = arith.constant 0 : index
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFpos_condEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFpos_condExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %i:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFpos_condEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFpos_condEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %iv = fir.load %i#0 : !fir.ref<i32>
+ %ic = fir.convert %iv : (i32) -> i64
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ // Conditional RMW: the unconditional init still dominates, so it promotes.
+ %cond = arith.cmpi sgt, %j, %c0 : index
+ fir.if %cond {
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xx#0 (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ }
+ return
+}
+
+// -----
+
+// a(:,i) = a(:,i) + x(j) (whole-section reduction with no constant-shape xx,
+// so the accumulator extent is a runtime value)
+// P4: shape is not a compile-time constant -> detected but not rewritten.
+// Kept descriptor-based since the extent must be a runtime value.
+// CHECK-LABEL: func.func @_QPneg_dynamic
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_dynamic(%arg0: !fir.box<!fir.array<?x?xf64>> {fir.bindc_name = "a"}, %arg1: !fir.box<!fir.array<?xf64>> {fir.bindc_name = "x"}, %arg2: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg3: !fir.ref<i32> {fir.bindc_name = "i"}, %arg4: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %1:2 = hlfir.declare %arg0 dummy_scope %0 arg 1 {uniq_name = "_QFneg_dynamicEa"} : (!fir.box<!fir.array<?x?xf64>>, !fir.dscope) -> (!fir.box<!fir.array<?x?xf64>>, !fir.box<!fir.array<?x?xf64>>)
+ %2:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFneg_dynamicEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %3 = fir.alloca i32 {bindc_name = "j", uniq_name = "_QFneg_dynamicEj"}
+ %4:2 = hlfir.declare %3 {uniq_name = "_QFneg_dynamicEj"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %5:2 = hlfir.declare %arg4 dummy_scope %0 arg 5 {uniq_name = "_QFneg_dynamicEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %6:2 = hlfir.declare %arg1 dummy_scope %0 arg 2 {uniq_name = "_QFneg_dynamicEx"} : (!fir.box<!fir.array<?xf64>>, !fir.dscope) -> (!fir.box<!fir.array<?xf64>>, !fir.box<!fir.array<?xf64>>)
+ %cst = arith.constant 0.000000e+00 : f64
+ %c1 = arith.constant 1 : index
+ %c0 = arith.constant 0 : index
+ %bd0:3 = fir.box_dims %1#1, %c0 : (!fir.box<!fir.array<?x?xf64>>, index) -> (index, index, index)
+ %il0 = fir.load %2#0 : !fir.ref<i32>
+ %ic0 = fir.convert %il0 : (i32) -> i64
+ %sh0 = fir.shape %bd0#1 : (index) -> !fir.shape<1>
+ %initd = hlfir.designate %1#0 (%c1:%bd0#1:%c1, %ic0) shape %sh0 : (!fir.box<!fir.array<?x?xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.box<!fir.array<?xf64>>
+ hlfir.assign %cst to %initd : f64, !fir.box<!fir.array<?xf64>>
+ %c1_i32 = arith.constant 1 : i32
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nl = fir.load %5#0 : !fir.ref<i32>
+ %ub = fir.convert %nl : (i32) -> index
+ %st = arith.constant 1 : index
+ %iv0 = fir.convert %lb : (index) -> i32
+ %loop = fir.do_loop %arg5 = %lb to %ub step %st iter_args(%arg6 = %iv0) -> (i32) {
+ fir.store %arg6 to %4#0 : !fir.ref<i32>
+ %c1_l = arith.constant 1 : index
+ %c0_l = arith.constant 0 : index
+ %bd:3 = fir.box_dims %1#1, %c0_l : (!fir.box<!fir.array<?x?xf64>>, index) -> (index, index, index)
+ %il = fir.load %2#0 : !fir.ref<i32>
+ %ic = fir.convert %il : (i32) -> i64
+ %sh = fir.shape %bd#1 : (index) -> !fir.shape<1>
+ %rd = hlfir.designate %1#0 (%c1_l:%bd#1:%c1_l, %ic) shape %sh : (!fir.box<!fir.array<?x?xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.box<!fir.array<?xf64>>
+ %jl = fir.load %4#0 : !fir.ref<i32>
+ %jc = fir.convert %jl : (i32) -> i64
+ %xd = hlfir.designate %6#0 (%jc) : (!fir.box<!fir.array<?xf64>>, i64) -> !fir.ref<f64>
+ %xv = fir.load %xd : !fir.ref<f64>
+ %e = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<?xf64> {
+ ^bb0(%k: index):
+ %r = hlfir.designate %rd (%k) : (!fir.box<!fir.array<?xf64>>, index) -> !fir.ref<f64>
+ %v = fir.load %r : !fir.ref<f64>
+ %s = arith.addf %v, %xv fastmath<contract> : f64
+ hlfir.yield_element %s : f64
+ }
+ %wd = hlfir.designate %1#0 (%c1_l:%bd#1:%c1_l, %ic) shape %sh : (!fir.box<!fir.array<?x?xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.box<!fir.array<?xf64>>
+ hlfir.assign %e to %wd : !hlfir.expr<?xf64>, !fir.box<!fir.array<?xf64>>
+ hlfir.destroy %e : !hlfir.expr<?xf64>
+ %ni = fir.convert %st : (index) -> i32
+ %nj = fir.load %4#0 : !fir.ref<i32>
+ %nn = arith.addi %nj, %ni overflow<nsw> : i32
+ fir.result %nn : i32
+ }
+ fir.store %loop to %4#0 : !fir.ref<i32>
+ return
+}
+
+// -----
+
+// a(:,i) = 0; do j; a(:,i) = a(:,i) + xx*x(j) (same source as @_QPpos, but
+// lowered to an unstructured CFG with the init and loop in separate blocks)
+// Promotes: the init still dominates the loop (cross-block dominance).
+// CHECK: #[[VEC:.*]] = #llvm.loop_vectorize<disable = false>
+// CHECK: #[[ANNOT:.*]] = #llvm.loop_annotation<vectorize = #[[VEC]]>
+// CHECK-LABEL: func.func @_QPpos_cfg
+// CHECK: fir.alloca !fir.array<3xf64>
+// CHECK: fir.do_loop {{.*}}attributes {loopAnnotation = #[[ANNOT]]}
+func.func @_QPpos_cfg(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFpos_cfgEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFpos_cfgExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %i:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFpos_cfgEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFpos_cfgEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %iv = fir.load %i#0 : !fir.ref<i32>
+ %ic = fir.convert %iv : (i32) -> i64
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ // Init and loop live in separate blocks: cross-block dominance still promotes.
+ cf.br ^bb1
+^bb1:
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xx#0 (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+
+// -----
+
+// a(:,i) = x(j) (broadcast; the RHS never reads a(:,i), so not a reduction)
+// Rejected: the RHS does not read the section, so this is not a read-modify-write.
+// CHECK-LABEL: func.func @_QPneg_notrmw
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_notrmw(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFneg_notrmwEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFneg_notrmwExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %i:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFneg_notrmwEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFneg_notrmwEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %iv = fir.load %i#0 : !fir.ref<i32>
+ %ic = fir.convert %iv : (i32) -> i64
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ %xe = hlfir.designate %xx#0 (%c1) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ // Broadcast: the RHS never reads a(:,i), so this is not a read-modify-write.
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ hlfir.yield_element %xv : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+
+// -----
+
+// a(1,i) = 0 a second in-loop write that may alias the section a(:,i).
+// P3 (writes): an aliasing in-loop assignment -> not promoted.
+// CHECK-LABEL: func.func @_QPneg_alias_write
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_alias_write(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFneg_alias_writeEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFneg_alias_writeExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %i:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFneg_alias_writeEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFneg_alias_writeEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %iv = fir.load %i#0 : !fir.ref<i32>
+ %ic = fir.convert %iv : (i32) -> i64
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ // An extra in-loop write a(1,i) aliases the section: P3-writes rejects.
+ %ivl0 = fir.load %i#0 : !fir.ref<i32>
+ %icl0 = fir.convert %ivl0 : (i32) -> i64
+ %aw = hlfir.designate %a#0 (%c1, %icl0) : (!fir.ref<!fir.array<3x4xf64>>, index, i64) -> !fir.ref<f64>
+ hlfir.assign %cst to %aw : f64, !fir.ref<f64>
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xx#0 (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+
+// -----
+
+// if (n > 0) a(:,i) = 0 (the init is conditional)
+// P2: the init does not dominate the loop -> not promoted.
+// CHECK-LABEL: func.func @_QPneg_cond_init
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_cond_init(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c0 = arith.constant 0 : index
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFneg_cond_initEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFneg_cond_initExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %i:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFneg_cond_initEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFneg_cond_initEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ // Conditional init: it does not dominate the loop, so P2 finds no init.
+ %n0 = fir.load %n#0 : !fir.ref<i32>
+ %n0i = fir.convert %n0 : (i32) -> index
+ %cond0 = arith.cmpi sgt, %n0i, %c0 : index
+ fir.if %cond0 {
+ %iv = fir.load %i#0 : !fir.ref<i32>
+ %ic = fir.convert %iv : (i32) -> i64
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ }
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xx#0 (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+
+// -----
+
+// a(:,i) = 0; do j; a(:,i) = a(:,i) + xx*x(j) then b(:,i) = 0; do j; b(:,i) = b(:,i) + xx*x(j)
+// Two independent reductions in one routine. Both promote: two temporaries,
+// two annotated loops.
+// CHECK: #[[VEC:.*]] = #llvm.loop_vectorize<disable = false>
+// CHECK: #[[ANNOT:.*]] = #llvm.loop_annotation<vectorize = #[[VEC]]>
+// CHECK-LABEL: func.func @_QPpos_multi
+// CHECK: fir.alloca !fir.array<3xf64>
+// CHECK: fir.alloca !fir.array<3xf64>
+// CHECK: fir.do_loop {{.*}}attributes {loopAnnotation = #[[ANNOT]]}
+// CHECK: fir.do_loop {{.*}}attributes {loopAnnotation = #[[ANNOT]]}
+func.func @_QPpos_multi(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}, %argb: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "b"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFpos_multiEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFpos_multiExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %i:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFpos_multiEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFpos_multiEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %b:2 = hlfir.declare %argb(%sh2) dummy_scope %0 arg 5 {uniq_name = "_QFpos_multiEb"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %iv = fir.load %i#0 : !fir.ref<i32>
+ %ic = fir.convert %iv : (i32) -> i64
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ // First reduction on a(:,i).
+ %inita = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %inita : f64, !fir.ref<!fir.array<3xf64>>
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ %rda = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %ela = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rda (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xx#0 (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wra = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %ela to %wra : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %ela : !hlfir.expr<3xf64>
+ }
+ // Second reduction on the independent array b(:,i).
+ %initb = hlfir.designate %b#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %initb : f64, !fir.ref<!fir.array<3xf64>>
+ fir.do_loop %jb = %lb to %ub step %c1 {
+ %ivlb = fir.load %i#0 : !fir.ref<i32>
+ %iclb = fir.convert %ivlb : (i32) -> i64
+ %rdb = hlfir.designate %b#0 (%c1:%c3:%c1, %iclb) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %elb = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%kb: index):
+ %be = hlfir.designate %rdb (%kb) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %bv = fir.load %be : !fir.ref<f64>
+ %xeb = hlfir.designate %xx#0 (%kb) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xvb = fir.load %xeb : !fir.ref<f64>
+ %sumb = arith.addf %bv, %xvb fastmath<contract> : f64
+ hlfir.yield_element %sumb : f64
+ }
+ %wrb = hlfir.designate %b#0 (%c1:%c3:%c1, %iclb) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %elb to %wrb : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %elb : !hlfir.expr<3xf64>
+ }
+ return
+}
+
+// -----
+
+// x(i) = a(1,i) read of the section between the init and the loop.
+// P2 (interval): the folded-away init would leave that read seeing stale
+// memory -> not promoted.
+// CHECK-LABEL: func.func @_QPneg_read_between
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_read_between(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}, %arg4: !fir.ref<f64> {fir.bindc_name = "res"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFneg_read_betweenEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFneg_read_betweenExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %i:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFneg_read_betweenEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFneg_read_betweenEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %res:2 = hlfir.declare %arg4 dummy_scope %0 arg 5 {uniq_name = "_QFneg_read_betweenEres"} : (!fir.ref<f64>, !fir.dscope) -> (!fir.ref<f64>, !fir.ref<f64>)
+ %iv = fir.load %i#0 : !fir.ref<i32>
+ %ic = fir.convert %iv : (i32) -> i64
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ // Read a(1,i) between the init and the loop: P2 (interval) rejects.
+ %rb_e = hlfir.designate %a#0 (%c1, %ic) : (!fir.ref<!fir.array<3x4xf64>>, index, i64) -> !fir.ref<f64>
+ %rb_v = fir.load %rb_e : !fir.ref<f64>
+ hlfir.assign %rb_v to %res#0 : f64, !fir.ref<f64>
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xx#0 (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+
+// -----
+
+// if (c) x(i) = a(1,i) a *conditional* read of the section between the init
+// and the loop. The read is dominated by the init but does not dominate the
+// loop, so a dominates-the-loop interval test would miss it.
+// P2 (interval): the folded-away init would leave that conditional read seeing
+// stale memory -> not promoted.
+// CHECK-LABEL: func.func @_QPneg_cond_read_between
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_cond_read_between(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}, %arg4: !fir.ref<f64> {fir.bindc_name = "res"}, %argc: !fir.ref<!fir.logical<4>> {fir.bindc_name = "c"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFneg_cond_read_betweenEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFneg_cond_read_betweenExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %i:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFneg_cond_read_betweenEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFneg_cond_read_betweenEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %res:2 = hlfir.declare %arg4 dummy_scope %0 arg 5 {uniq_name = "_QFneg_cond_read_betweenEres"} : (!fir.ref<f64>, !fir.dscope) -> (!fir.ref<f64>, !fir.ref<f64>)
+ %c:2 = hlfir.declare %argc dummy_scope %0 arg 6 {uniq_name = "_QFneg_cond_read_betweenEc"} : (!fir.ref<!fir.logical<4>>, !fir.dscope) -> (!fir.ref<!fir.logical<4>>, !fir.ref<!fir.logical<4>>)
+ %iv = fir.load %i#0 : !fir.ref<i32>
+ %ic = fir.convert %iv : (i32) -> i64
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ // Conditional read a(1,i) between the init and the loop: dominated by the
+ // init but not dominating the loop, so P2 (interval) must still reject.
+ %cl = fir.load %c#0 : !fir.ref<!fir.logical<4>>
+ %cb = fir.convert %cl : (!fir.logical<4>) -> i1
+ fir.if %cb {
+ %rb_e = hlfir.designate %a#0 (%c1, %ic) : (!fir.ref<!fir.array<3x4xf64>>, index, i64) -> !fir.ref<f64>
+ %rb_v = fir.load %rb_e : !fir.ref<f64>
+ hlfir.assign %rb_v to %res#0 : f64, !fir.ref<f64>
+ }
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xx#0 (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+
+// -----
+
+// a(1,i) = 0 write to the section between the init and the loop.
+// P2 (interval): the post-loop copy-out would clobber that write -> not promoted.
+// CHECK-LABEL: func.func @_QPneg_write_between
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_write_between(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFneg_write_betweenEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFneg_write_betweenExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %i:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFneg_write_betweenEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFneg_write_betweenEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %iv = fir.load %i#0 : !fir.ref<i32>
+ %ic = fir.convert %iv : (i32) -> i64
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ // Write a(1,i) between the init and the loop: the post-loop copy-out would
+ // clobber it, so P2 (interval) rejects.
+ %wb_e = hlfir.designate %a#0 (%c1, %ic) : (!fir.ref<!fir.array<3x4xf64>>, index, i64) -> !fir.ref<f64>
+ hlfir.assign %cst to %wb_e : f64, !fir.ref<f64>
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xx#0 (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+
+// -----
+
+// i = i + 1 the section index is reassigned between the init and the loop, so
+// the init defines a(:,i0) but the loop reduces a(:,i0+1).
+// P2 (interval): a subscript the section reads is rewritten -> not promoted.
+// CHECK-LABEL: func.func @_QPneg_index_between
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_index_between(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFneg_index_betweenEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFneg_index_betweenExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %i:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFneg_index_betweenEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFneg_index_betweenEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %iv = fir.load %i#0 : !fir.ref<i32>
+ %ic = fir.convert %iv : (i32) -> i64
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ // Reassign the section index i between the init and the loop: P2 (interval)
+ // sees a subscript the section reads rewritten, so it rejects.
+ %ib_v = fir.load %i#0 : !fir.ref<i32>
+ %ib_n = arith.addi %ib_v, %c1_i32 : i32
+ hlfir.assign %ib_n to %i#0 : i32, !fir.ref<i32>
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xx#0 (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+
+// -----
+
+// a(:,i) = 0; do j; a(:,i) = a(:,i) + xx*x(j) (twice, on the SAME section a(:,i))
+// Both must promote: findDominatingFullDef must pick each loop's own (nearest)
+// init, not the prior loop's copy-out.
+// CHECK: #[[VEC:.*]] = #llvm.loop_vectorize<disable = false>
+// CHECK: #[[ANNOT:.*]] = #llvm.loop_annotation<vectorize = #[[VEC]]>
+// CHECK-LABEL: func.func @_QPpos_multi_same
+// CHECK: fir.alloca !fir.array<3xf64>
+// CHECK: fir.alloca !fir.array<3xf64>
+// CHECK: fir.do_loop {{.*}}attributes {loopAnnotation = #[[ANNOT]]}
+// CHECK: fir.do_loop {{.*}}attributes {loopAnnotation = #[[ANNOT]]}
+func.func @_QPpos_multi_same(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFpos_multi_sameEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFpos_multi_sameExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %i:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFpos_multi_sameEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFpos_multi_sameEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %iv = fir.load %i#0 : !fir.ref<i32>
+ %ic = fir.convert %iv : (i32) -> i64
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ // First reduction on a(:,i).
+ %inita = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %inita : f64, !fir.ref<!fir.array<3xf64>>
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ %rda = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %ela = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rda (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xx#0 (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wra = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %ela to %wra : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %ela : !hlfir.expr<3xf64>
+ }
+ // Second reduction on the same a(:,i): each loop must pick its own nearest init.
+ %inits = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %inits : f64, !fir.ref<!fir.array<3xf64>>
+ fir.do_loop %js = %lb to %ub step %c1 {
+ %ivls = fir.load %i#0 : !fir.ref<i32>
+ %icls = fir.convert %ivls : (i32) -> i64
+ %rds = hlfir.designate %a#0 (%c1:%c3:%c1, %icls) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %els = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%ks: index):
+ %aes = hlfir.designate %rds (%ks) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %avs = fir.load %aes : !fir.ref<f64>
+ %xes = hlfir.designate %xx#0 (%ks) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xvs = fir.load %xes : !fir.ref<f64>
+ %sums = arith.addf %avs, %xvs fastmath<contract> : f64
+ hlfir.yield_element %sums : f64
+ }
+ %wrs = hlfir.designate %a#0 (%c1:%c3:%c1, %icls) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %els to %wrs : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %els : !hlfir.expr<3xf64>
+ }
+ return
+}
+
+// -----
+
+// a(:,i) = a(:,i) + xx*x(j) (canonical reduction, as flang lowers it at O2)
+// One designate is both the RMW target and the RHS read base (the shape the
+// front end emits after CSE), so the read's users include the RMW assign. Must
+// still promote AND annotate -- regression guard for the vectorize-hint gate.
+// CHECK: #[[VEC:.*]] = #llvm.loop_vectorize<disable = false>
+// CHECK: #[[ANNOT:.*]] = #llvm.loop_annotation<vectorize = #[[VEC]]>
+// CHECK-LABEL: func.func @_QPpos_shared
+// CHECK: fir.alloca !fir.array<3xf64>
+// CHECK: fir.do_loop {{.*}}attributes {loopAnnotation = #[[ANNOT]]}
+func.func @_QPpos_shared(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFpos_sharedEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFpos_sharedExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %i:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFpos_sharedEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFpos_sharedEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %iv = fir.load %i#0 : !fir.ref<i32>
+ %ic = fir.convert %iv : (i32) -> i64
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ // One designate is both the RMW read base and the write target (CSE'd).
+ %rw = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rw (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xx#0 (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ hlfir.assign %el to %rw : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+
+// -----
+
+// a(:,i) = 0; do k; do j; a(:,i) = a(:,i) + x(j); end; end; y = a(1,i)
+// The reduction is in the inner loop but the init dominates the OUTER loop, so
+// (the post-loop read blocks the inner match) it is promoted at the outer loop.
+// The vectorize hint must still land on the inner loop, which is what vectorizes.
+// CHECK: #[[VEC:.*]] = #llvm.loop_vectorize<disable = false>
+// CHECK: #[[ANNOT:.*]] = #llvm.loop_annotation<vectorize = #[[VEC]]>
+// CHECK-LABEL: func.func @_QPpos_nested
+// CHECK: fir.alloca !fir.array<3xf64>
+// CHECK: fir.do_loop
+// CHECK-NOT: loopAnnotation
+// CHECK: fir.do_loop {{.*}}attributes {loopAnnotation = #[[ANNOT]]}
+func.func @_QPpos_nested(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}, %argm: !fir.ref<i32> {fir.bindc_name = "m"}, %argr: !fir.ref<f64> {fir.bindc_name = "res"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFpos_nestedEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFpos_nestedExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %i:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFpos_nestedEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFpos_nestedEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %m:2 = hlfir.declare %argm dummy_scope %0 arg 5 {uniq_name = "_QFpos_nestedEm"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %res:2 = hlfir.declare %argr dummy_scope %0 arg 6 {uniq_name = "_QFpos_nestedEres"} : (!fir.ref<f64>, !fir.dscope) -> (!fir.ref<f64>, !fir.ref<f64>)
+ %iv = fir.load %i#0 : !fir.ref<i32>
+ %ic = fir.convert %iv : (i32) -> i64
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %ml = fir.load %m#0 : !fir.ref<i32>
+ %ubk = fir.convert %ml : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ubj = fir.convert %nv : (i32) -> index
+ // Reduction in the inner loop; init dominates the outer loop, so it promotes
+ // at the outer loop while the vectorize hint lands on the inner loop.
+ fir.do_loop %k = %lb to %ubk step %c1 {
+ fir.do_loop %j = %lb to %ubj step %c1 {
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%kk: index):
+ %ae = hlfir.designate %rd (%kk) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xx#0 (%kk) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ }
+ // Post-outer-loop read a(1,i) -> res: forces the match at the outer loop.
+ %pr_iv = fir.load %i#0 : !fir.ref<i32>
+ %pr_ic = fir.convert %pr_iv : (i32) -> i64
+ %pr_e = hlfir.designate %a#0 (%c1, %pr_ic) : (!fir.ref<!fir.array<3x4xf64>>, index, i64) -> !fir.ref<f64>
+ %pr_v = fir.load %pr_e : !fir.ref<f64>
+ hlfir.assign %pr_v to %res#0 : f64, !fir.ref<f64>
+ return
+}
+
+// -----
+
+// a(:,i) = a(:,i) + sum(a(:,i)) (the RHS reduces the whole section; sum() is
+// inlined to a nested loop, as SimplifyHLFIRIntrinsics does at O2). Promoted,
+// but NOT annotated: the nested loop reads the accumulator (a coupled
+// recurrence), so the reduction loop is not vectorizable.
+// CHECK-LABEL: func.func @_QPnovec_reduce
+// CHECK: fir.alloca !fir.array<3xf64>
+// CHECK-NOT: loopAnnotation
+func.func @_QPnovec_reduce(%arg0: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh) dummy_scope %0 arg 1 {uniq_name = "_QFnovec_reduceEa"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %n:2 = hlfir.declare %arg1 dummy_scope %0 arg 2 {uniq_name = "_QFnovec_reduceEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ // Inlined sum reads the accumulator (a coupled recurrence): promoted but
+ // the reduction loop is not vectorizable, so it is left un-annotated.
+ %c0f = arith.constant 0.000000e+00 : f64
+ %s = fir.do_loop %ksum = %c1 to %c3 step %c1 iter_args(%acc = %c0f) -> (f64) {
+ %rsum = hlfir.designate %rd (%ksum) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %vsum = fir.load %rsum : !fir.ref<f64>
+ %nacc = arith.addf %acc, %vsum fastmath<contract> : f64
+ fir.result %nacc : f64
+ }
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %sum = arith.addf %av, %s fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+
+// -----
+
+// a(:,i) = 0; do j; s = 0; do k=1,3; s = s + xx(k); end; a(:,i) = a(:,i) + s
+// The inner loop is a small constant-trip scratch loop that reads a local xx,
+// not the accumulator, so it is fully unrolled before vectorization. Promoted
+// AND annotated: a nested loop only blocks the hint if it reads the accumulator
+// or is not fully unrollable.
+// CHECK: #[[VEC:.*]] = #llvm.loop_vectorize<disable = false>
+// CHECK: #[[ANNOT:.*]] = #llvm.loop_annotation<vectorize = #[[VEC]]>
+// CHECK-LABEL: func.func @_QPpos_scratch_loop
+// CHECK: fir.alloca !fir.array<3xf64>
+// CHECK: fir.do_loop {{.*}}attributes {loopAnnotation = #[[ANNOT]]}
+func.func @_QPpos_scratch_loop(%arg0: !fir.ref<!fir.array<3x4xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<i32> {fir.bindc_name = "i"}, %arg2: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c4 = arith.constant 4 : index
+ %c1_i32 = arith.constant 1 : i32
+ %cst = arith.constant 0.000000e+00 : f64
+ %c0f = arith.constant 0.000000e+00 : f64
+ %sh2 = fir.shape %c3, %c4 : (index, index) -> !fir.shape<2>
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh2) dummy_scope %0 arg 1 {uniq_name = "_QFpos_scratch_loopEa"} : (!fir.ref<!fir.array<3x4xf64>>, !fir.shape<2>, !fir.dscope) -> (!fir.ref<!fir.array<3x4xf64>>, !fir.ref<!fir.array<3x4xf64>>)
+ %i:2 = hlfir.declare %arg1 dummy_scope %0 arg 2 {uniq_name = "_QFpos_scratch_loopEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %n:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFpos_scratch_loopEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %xx = fir.alloca !fir.array<3xf64> {bindc_name = "xx", uniq_name = "_QFpos_scratch_loopExx"}
+ %xxd:2 = hlfir.declare %xx(%sh) {uniq_name = "_QFpos_scratch_loopExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %iv = fir.load %i#0 : !fir.ref<i32>
+ %ic = fir.convert %iv : (i32) -> i64
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1, %ic) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %ivl = fir.load %i#0 : !fir.ref<i32>
+ %icl = fir.convert %ivl : (i32) -> i64
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ // Small constant-trip scratch loop over the local xx, not the accumulator:
+ // fully unrolled before vectorization, so it does not block the hint.
+ %s = fir.do_loop %ksum = %c1 to %c3 step %c1 iter_args(%acc = %c0f) -> (f64) {
+ %rsum = hlfir.designate %xxd#0 (%ksum) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %vsum = fir.load %rsum : !fir.ref<f64>
+ %nacc = arith.addf %acc, %vsum fastmath<contract> : f64
+ fir.result %nacc : f64
+ }
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %sum = arith.addf %av, %s fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1, %icl) shape %sh : (!fir.ref<!fir.array<3x4xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+
+// -----
+
+// type(pt) :: a(:,:); do j; a(:,i) = a(:,i) (derived-type section)
+// The element type is not a trivial value type, so the rewrite (stack temporary
+// + value copy) does not apply. P0 is checked before the dominating-init lookup,
+// so no init is needed to reject.
+// P0: non-trivial element type -> not promoted.
+// CHECK-LABEL: func.func @_QPneg_derived
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_derived(%arg0: !fir.ref<!fir.array<3x!fir.type<_QTpt{x:f64}>>> {fir.bindc_name = "a"}, %arg1: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c1_i32 = arith.constant 1 : i32
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh) dummy_scope %0 arg 1 {uniq_name = "_QFneg_derivedEa"} : (!fir.ref<!fir.array<3x!fir.type<_QTpt{x:f64}>>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3x!fir.type<_QTpt{x:f64}>>>, !fir.ref<!fir.array<3x!fir.type<_QTpt{x:f64}>>>)
+ %n:2 = hlfir.declare %arg1 dummy_scope %0 arg 2 {uniq_name = "_QFneg_derivedEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3x!fir.type<_QTpt{x:f64}>>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3x!fir.type<_QTpt{x:f64}>>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3x!fir.type<_QTpt{x:f64}>> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3x!fir.type<_QTpt{x:f64}>>>, index) -> !fir.ref<!fir.type<_QTpt{x:f64}>>
+ %av = fir.load %ae : !fir.ref<!fir.type<_QTpt{x:f64}>>
+ hlfir.yield_element %av : !fir.type<_QTpt{x:f64}>
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3x!fir.type<_QTpt{x:f64}>>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3x!fir.type<_QTpt{x:f64}>>>
+ hlfir.assign %el to %wr : !hlfir.expr<3x!fir.type<_QTpt{x:f64}>>, !fir.ref<!fir.array<3x!fir.type<_QTpt{x:f64}>>>
+ hlfir.destroy %el : !hlfir.expr<3x!fir.type<_QTpt{x:f64}>>
+ }
+ return
+}
+
+// -----
+
+// real(8), volatile :: a(:,:); do j; a(:,i) = a(:,i) (volatile section)
+// The element type is trivial, so the trivial-type check passes and the
+// separate volatility check rejects. Like @_QPneg_derived, P0 is checked before
+// the dominating-init lookup, so no init is needed to reject.
+// P0: volatile section -> not promoted.
+// CHECK-LABEL: func.func @_QPneg_volatile
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_volatile(%arg0: !fir.ref<!fir.array<3xf64>, volatile> {fir.bindc_name = "a"}, %arg1: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c1 = arith.constant 1 : index
+ %c3 = arith.constant 3 : index
+ %c1_i32 = arith.constant 1 : i32
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh) dummy_scope %0 arg 1 {fortran_attrs = #fir.var_attrs<volatile>, uniq_name = "_QFneg_volatileEa"} : (!fir.ref<!fir.array<3xf64>, volatile>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>, volatile>, !fir.ref<!fir.array<3xf64>, volatile>)
+ %n:2 = hlfir.declare %arg1 dummy_scope %0 arg 2 {uniq_name = "_QFneg_volatileEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>, volatile>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>, volatile>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>, volatile>, index) -> !fir.ref<f64, volatile>
+ %av = fir.load %ae : !fir.ref<f64, volatile>
+ hlfir.yield_element %av : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>, volatile>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>, volatile>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>, volatile>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+
+// -----
+
+// !$omp parallel do private(i,j): a(:,i) = 0; do j; a(:,i) = a(:,i) + xx. The
+// section index i and the inner counter j are OpenMP privates. P1
+// must keep the hlfir.declare address so AliasAnalysis classifies i as a
+// private and does not treat the in-loop store to j as rewriting i. Promoted
+// and annotated. The omp.private decls stay in this chunk.
+// CHECK: #[[VEC:.*]] = #llvm.loop_vectorize<disable = false>
+// CHECK: #[[ANNOT:.*]] = #llvm.loop_annotation<vectorize = #[[VEC]]>
+// CHECK-LABEL: func.func @_QPompred
+// CHECK: fir.alloca !fir.array<3xf64>
+// CHECK: fir.do_loop {{.*}}attributes {loopAnnotation = #[[ANNOT]]}
+omp.private {type = private} @_QFompredEi_private_i32 : i32
+omp.private {type = private} @_QFompredEj_private_i32 : i32
+func.func @_QPompred(%arg0: !fir.box<!fir.array<?x?xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "i"}, %arg3: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %1:2 = hlfir.declare %arg0 dummy_scope %0 {uniq_name = "_QFompredEa"} : (!fir.box<!fir.array<?x?xf64>>, !fir.dscope) -> (!fir.box<!fir.array<?x?xf64>>, !fir.box<!fir.array<?x?xf64>>)
+ %2:2 = hlfir.declare %arg2 dummy_scope %0 {uniq_name = "_QFompredEi"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %3:2 = hlfir.declare %arg3 dummy_scope %0 {uniq_name = "_QFompredEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %c3 = arith.constant 3 : index
+ %sh3 = fir.shape %c3 : (index) -> !fir.shape<1>
+ %4:2 = hlfir.declare %arg1(%sh3) dummy_scope %0 {uniq_name = "_QFompredExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %jalloc = fir.alloca i32 {bindc_name = "j", uniq_name = "_QFompredEj"}
+ %5:2 = hlfir.declare %jalloc {uniq_name = "_QFompredEj"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %c0_i32 = arith.constant 0 : i32
+ %nload = fir.load %3#0 : !fir.ref<i32>
+ %c1_i32 = arith.constant 1 : i32
+ %ubi = arith.subi %nload, %c1_i32 : i32
+ omp.parallel {
+ omp.wsloop private(@_QFompredEi_private_i32 %2#0 -> %argi, @_QFompredEj_private_i32 %5#0 -> %argj : !fir.ref<i32>, !fir.ref<i32>) {
+ omp.loop_nest (%iv) : i32 = (%c0_i32) to (%ubi) inclusive step (%c1_i32) {
+ %pi:2 = hlfir.declare %argi {uniq_name = "_QFompredEi"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %pj:2 = hlfir.declare %argj {uniq_name = "_QFompredEj"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+ hlfir.assign %iv to %pi#0 : i32, !fir.ref<i32>
+ %cst = arith.constant 0.000000e+00 : f64
+ %c1 = arith.constant 1 : index
+ %c0 = arith.constant 0 : index
+ %bd0:3 = fir.box_dims %1#1, %c0 : (!fir.box<!fir.array<?x?xf64>>, index) -> (index, index, index)
+ %il0 = fir.load %pi#0 : !fir.ref<i32>
+ %ic0 = fir.convert %il0 : (i32) -> i64
+ %initd = hlfir.designate %1#0 (%c1:%bd0#1:%c1, %ic0) shape %sh3 : (!fir.box<!fir.array<?x?xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.box<!fir.array<?xf64>>
+ hlfir.assign %cst to %initd : f64, !fir.box<!fir.array<?xf64>>
+ %lb = arith.constant 1 : index
+ %nl = fir.load %3#0 : !fir.ref<i32>
+ %ubj = fir.convert %nl : (i32) -> index
+ %st = arith.constant 1 : index
+ %jiv0 = fir.convert %lb : (index) -> i32
+ %jloop = fir.do_loop %arg5 = %lb to %ubj step %st iter_args(%arg6 = %jiv0) -> (i32) {
+ fir.store %arg6 to %pj#0 : !fir.ref<i32>
+ %c1_l = arith.constant 1 : index
+ %c0_l = arith.constant 0 : index
+ %bd:3 = fir.box_dims %1#1, %c0_l : (!fir.box<!fir.array<?x?xf64>>, index) -> (index, index, index)
+ %il = fir.load %pi#0 : !fir.ref<i32>
+ %ic = fir.convert %il : (i32) -> i64
+ %rd = hlfir.designate %1#0 (%c1_l:%bd#1:%c1_l, %ic) shape %sh3 : (!fir.box<!fir.array<?x?xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.box<!fir.array<?xf64>>
+ %xxsl = hlfir.designate %4#0 (%c1_l:%c3:%c1_l) shape %sh3 : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %e = hlfir.elemental %sh3 unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %r = hlfir.designate %rd (%k) : (!fir.box<!fir.array<?xf64>>, index) -> !fir.ref<f64>
+ %xr = hlfir.designate %xxsl (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %v = fir.load %r : !fir.ref<f64>
+ %xv = fir.load %xr : !fir.ref<f64>
+ %sum = arith.addf %v, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wd = hlfir.designate %1#0 (%c1_l:%bd#1:%c1_l, %ic) shape %sh3 : (!fir.box<!fir.array<?x?xf64>>, index, index, index, i64, !fir.shape<1>) -> !fir.box<!fir.array<?xf64>>
+ hlfir.assign %e to %wd : !hlfir.expr<3xf64>, !fir.box<!fir.array<?xf64>>
+ hlfir.destroy %e : !hlfir.expr<3xf64>
+ %ni = fir.convert %st : (index) -> i32
+ %nj = fir.load %pj#0 : !fir.ref<i32>
+ %nn = arith.addi %nj, %ni overflow<nsw> : i32
+ fir.result %nn : i32
+ }
+ fir.store %jloop to %pj#0 : !fir.ref<i32>
+ omp.yield
+ }
+ }
+ omp.terminator
+ }
+ return
+}
+
+// -----
+
+// @_QPneg_direct_store: an extra direct element write into the section via
+// fir.array_coor + fir.store (not an hlfir.assign) inside the loop. P3-writes
+// must reject -- promotion redirects the reduction to T while the direct store
+// still updates a(:), so the post-loop copy-out would clobber it. Source:
+// real(8) a(3), xx(3); a = 0; do j=1,n; a = a + xx; a(2) = 1.0; end do
+// CHECK-LABEL: func.func @_QPneg_direct_store
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_direct_store(%arg0: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c3 = arith.constant 3 : index
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh) dummy_scope %0 arg 1 {uniq_name = "_QFneg_direct_storeEa"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFneg_direct_storeExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %n:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFneg_direct_storeEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %cst = arith.constant 0.000000e+00 : f64
+ %c1 = arith.constant 1 : index
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ %c1_i32 = arith.constant 1 : i32
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %xxs = hlfir.designate %xx#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xxs (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ %c2 = arith.constant 2 : index
+ %elt = fir.array_coor %a#0(%sh) %c2 : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, index) -> !fir.ref<f64>
+ %one = arith.constant 1.000000e+00 : f64
+ fir.store %one to %elt : !fir.ref<f64>
+ }
+ return
+}
+
+// -----
+
+// @_QPneg_direct_read: a direct element read of the section via fir.array_coor
+// + fir.load (not an hlfir.designate) inside the loop. P3-reads must reject --
+// after promotion the reduction accumulates into T and the section is not
+// updated in the loop, so the direct read would observe stale memory. Source:
+// real(8) a(3), xx(3), res; a = 0; do j=1,n; a = a + xx; res = a(2); end do
+// CHECK-LABEL: func.func @_QPneg_direct_read
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_direct_read(%arg0: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "n"}, %arg3: !fir.ref<f64> {fir.bindc_name = "res"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c3 = arith.constant 3 : index
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh) dummy_scope %0 arg 1 {uniq_name = "_QFneg_direct_readEa"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFneg_direct_readExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %n:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFneg_direct_readEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %res:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFneg_direct_readEres"} : (!fir.ref<f64>, !fir.dscope) -> (!fir.ref<f64>, !fir.ref<f64>)
+ %cst = arith.constant 0.000000e+00 : f64
+ %c1 = arith.constant 1 : index
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ %c1_i32 = arith.constant 1 : i32
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %xxs = hlfir.designate %xx#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xxs (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ %c2 = arith.constant 2 : index
+ %elt = fir.array_coor %a#0(%sh) %c2 : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, index) -> !fir.ref<f64>
+ %v = fir.load %elt : !fir.ref<f64>
+ fir.store %v to %res#0 : !fir.ref<f64>
+ }
+ return
+}
+
+// -----
+
+// @_QPpos_reused_read: the section read designator %shared is loop-invariant
+// (defined before the loop) and reused after the loop, mirroring CSE'd input
+// where one a(:) designator is shared. Promotion must redirect only the in-loop
+// use to T; the post-loop use must keep reading the copied-out section, not the
+// stale temporary. Source: real(8) a(3), xx(3), res;
+// a = 0; do j=1,n; a = a + xx; end do; a = 99.0; res = a(2)
+// CHECK-LABEL: func.func @_QPpos_reused_read
+// CHECK: %[[T:.*]] = fir.alloca !fir.array<3xf64>
+// CHECK: %[[SHARED:.*]] = hlfir.designate %{{.*}}#0 {{.*}} -> !fir.ref<!fir.array<3xf64>>
+// CHECK: hlfir.designate %[[T]] (%{{.*}}) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+// CHECK: hlfir.designate %[[SHARED]] (%{{.*}}) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+func.func @_QPpos_reused_read(%arg0: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "n"}, %arg3: !fir.ref<f64> {fir.bindc_name = "res"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c3 = arith.constant 3 : index
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh) dummy_scope %0 arg 1 {uniq_name = "_QFpos_reused_readEa"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFpos_reused_readExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %n:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFpos_reused_readEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %res:2 = hlfir.declare %arg3 dummy_scope %0 arg 4 {uniq_name = "_QFpos_reused_readEres"} : (!fir.ref<f64>, !fir.dscope) -> (!fir.ref<f64>, !fir.ref<f64>)
+ %cst = arith.constant 0.000000e+00 : f64
+ %c1 = arith.constant 1 : index
+ %shared = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ %c1_i32 = arith.constant 1 : i32
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %xxs = hlfir.designate %xx#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %shared (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xxs (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ %c99 = arith.constant 9.900000e+01 : f64
+ %w2 = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %c99 to %w2 : f64, !fir.ref<!fir.array<3xf64>>
+ %c2 = arith.constant 2 : index
+ %e2 = hlfir.designate %shared (%c2) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %v = fir.load %e2 : !fir.ref<f64>
+ fir.store %v to %res#0 : !fir.ref<f64>
+ return
+}
+
+// -----
+
+// @_QPpos_novector: the loop already carries a NOVECTOR annotation (vectorize
+// disable = true). Promotion still applies, but the vectorize hint must not
+// override the user's directive -- the annotation is left unchanged.
+// CHECK: #[[NVEC:.*]] = #llvm.loop_vectorize<disable = true>
+// CHECK: #[[NOVEC_ANNO:.*]] = #llvm.loop_annotation<vectorize = #[[NVEC]]>
+// CHECK-LABEL: func.func @_QPpos_novector
+// CHECK: fir.alloca !fir.array<3xf64>
+// CHECK: fir.do_loop {{.*}}attributes {loopAnnotation = #[[NOVEC_ANNO]]}
+// NOVEC-LABEL: func.func @_QPpos_novector
+// NOVEC: fir.alloca !fir.array<3xf64>
+// NOVEC: fir.do_loop {{.*}}loopAnnotation
+func.func @_QPpos_novector(%arg0: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c3 = arith.constant 3 : index
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh) dummy_scope %0 arg 1 {uniq_name = "_QFpos_novectorEa"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFpos_novectorExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %n:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFpos_novectorEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %cst = arith.constant 0.000000e+00 : f64
+ %c1 = arith.constant 1 : index
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ %c1_i32 = arith.constant 1 : i32
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 attributes {loopAnnotation = #llvm.loop_annotation<vectorize = #llvm.loop_vectorize<disable = true>>} {
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %xxs = hlfir.designate %xx#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xxs (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+
+// -----
+
+// @_QPpos_preserve: the loop already carries an annotation with an unrelated
+// field (mustProgress). Promotion sets only the vectorize-enable field and
+// keeps every other field.
+// CHECK: #[[VEC:.*]] = #llvm.loop_vectorize<disable = false>
+// CHECK: #[[PRES:.*]] = #llvm.loop_annotation<vectorize = #[[VEC]], mustProgress = true>
+// CHECK-LABEL: func.func @_QPpos_preserve
+// CHECK: fir.alloca !fir.array<3xf64>
+// CHECK: fir.do_loop {{.*}}attributes {loopAnnotation = #[[PRES]]}
+// NOVEC-LABEL: func.func @_QPpos_preserve
+// NOVEC: fir.alloca !fir.array<3xf64>
+// NOVEC: fir.do_loop {{.*}}loopAnnotation
+func.func @_QPpos_preserve(%arg0: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "n"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c3 = arith.constant 3 : index
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh) dummy_scope %0 arg 1 {uniq_name = "_QFpos_preserveEa"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFpos_preserveExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %n:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFpos_preserveEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %cst = arith.constant 0.000000e+00 : f64
+ %c1 = arith.constant 1 : index
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ %c1_i32 = arith.constant 1 : i32
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 attributes {loopAnnotation = #llvm.loop_annotation<mustProgress = true>} {
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %xxs = hlfir.designate %xx#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xxs (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+
+// -----
+
+// @_QPneg_fir_copy: a fir.copy reads the whole section (its source) into
+// another array inside the loop. fir.copy exposes the read via
+// MemoryEffectOpInterface, not hlfir.designate/fir.load, so P3-reads must
+// reject it -- P3-writes passes because the copy's write targets the
+// non-aliasing destination. Source: real(8) a(3), xx(3), other(3);
+// a = 0; do j=1,n; a = a + xx; other = a; end do
+// CHECK-LABEL: func.func @_QPneg_fir_copy
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_fir_copy(%arg0: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "n"}, %arg3: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "other"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c3 = arith.constant 3 : index
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh) dummy_scope %0 arg 1 {uniq_name = "_QFneg_fir_copyEa"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFneg_fir_copyExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %n:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFneg_fir_copyEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %other:2 = hlfir.declare %arg3(%sh) dummy_scope %0 arg 4 {uniq_name = "_QFneg_fir_copyEother"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %cst = arith.constant 0.000000e+00 : f64
+ %c1 = arith.constant 1 : index
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ %c1_i32 = arith.constant 1 : i32
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %xxs = hlfir.designate %xx#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xxs (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ fir.copy %a#0 to %other#0 : !fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>
+ }
+ return
+}
+
+// -----
+
+// @_QPneg_copy_between: a fir.copy reads the whole section (its source) into
+// another array *between* the init and the loop. Promotion folds the init away
+// and only copies the reduction back after the loop, so this copy would read
+// stale (uninitialized) section memory. P2 (interval) must reject it via the
+// generic memory-effects read check, since fir.copy is neither an
+// hlfir.designate nor a fir.load. Source: real(8) a(3), xx(3), other(3);
+// a = 0; other = a; do j=1,n; a = a + xx; end do
+// CHECK-LABEL: func.func @_QPneg_copy_between
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_copy_between(%arg0: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "n"}, %arg3: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "other"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c3 = arith.constant 3 : index
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh) dummy_scope %0 arg 1 {uniq_name = "_QFneg_copy_betweenEa"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFneg_copy_betweenExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %n:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFneg_copy_betweenEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %other:2 = hlfir.declare %arg3(%sh) dummy_scope %0 arg 4 {uniq_name = "_QFneg_copy_betweenEother"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %cst = arith.constant 0.000000e+00 : f64
+ %c1 = arith.constant 1 : index
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ // Read the whole section between the init and the loop via fir.copy.
+ fir.copy %a#0 to %other#0 : !fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>
+ %c1_i32 = arith.constant 1 : i32
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %xxs = hlfir.designate %xx#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xxs (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+
+// -----
+
+// @_QPneg_assign_between: like @_QPneg_copy_between, but the copy other = a is
+// an hlfir.assign whose RHS designates the section (not a fir.copy). The LHS
+// (other) does not alias the section, so P2's hlfir.assign special case must
+// also check the RHS -- the section read between the init and the loop would
+// see stale storage once the init is folded into the temporary. Source:
+// real(8) a(3), xx(3), other(3); a = 0; other = a; do j=1,n; a = a + xx; end do
+// CHECK-LABEL: func.func @_QPneg_assign_between
+// CHECK-NOT: fir.alloca !fir.array
+// CHECK-NOT: loopAnnotation
+func.func @_QPneg_assign_between(%arg0: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "a"}, %arg1: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "xx"}, %arg2: !fir.ref<i32> {fir.bindc_name = "n"}, %arg3: !fir.ref<!fir.array<3xf64>> {fir.bindc_name = "other"}) {
+ %0 = fir.dummy_scope : !fir.dscope
+ %c3 = arith.constant 3 : index
+ %sh = fir.shape %c3 : (index) -> !fir.shape<1>
+ %a:2 = hlfir.declare %arg0(%sh) dummy_scope %0 arg 1 {uniq_name = "_QFneg_assign_betweenEa"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %xx:2 = hlfir.declare %arg1(%sh) dummy_scope %0 arg 2 {uniq_name = "_QFneg_assign_betweenExx"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %n:2 = hlfir.declare %arg2 dummy_scope %0 arg 3 {uniq_name = "_QFneg_assign_betweenEn"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+ %other:2 = hlfir.declare %arg3(%sh) dummy_scope %0 arg 4 {uniq_name = "_QFneg_assign_betweenEother"} : (!fir.ref<!fir.array<3xf64>>, !fir.shape<1>, !fir.dscope) -> (!fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>)
+ %cst = arith.constant 0.000000e+00 : f64
+ %c1 = arith.constant 1 : index
+ %init = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %cst to %init : f64, !fir.ref<!fir.array<3xf64>>
+ // Read the whole section between the init and the loop via an hlfir.assign
+ // whose RHS designates the section.
+ %rb = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %rb to %other#0 : !fir.ref<!fir.array<3xf64>>, !fir.ref<!fir.array<3xf64>>
+ %c1_i32 = arith.constant 1 : i32
+ %lb = fir.convert %c1_i32 : (i32) -> index
+ %nv = fir.load %n#0 : !fir.ref<i32>
+ %ub = fir.convert %nv : (i32) -> index
+ fir.do_loop %j = %lb to %ub step %c1 {
+ %rd = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %xxs = hlfir.designate %xx#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ %el = hlfir.elemental %sh unordered : (!fir.shape<1>) -> !hlfir.expr<3xf64> {
+ ^bb0(%k: index):
+ %ae = hlfir.designate %rd (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %av = fir.load %ae : !fir.ref<f64>
+ %xe = hlfir.designate %xxs (%k) : (!fir.ref<!fir.array<3xf64>>, index) -> !fir.ref<f64>
+ %xv = fir.load %xe : !fir.ref<f64>
+ %sum = arith.addf %av, %xv fastmath<contract> : f64
+ hlfir.yield_element %sum : f64
+ }
+ %wr = hlfir.designate %a#0 (%c1:%c3:%c1) shape %sh : (!fir.ref<!fir.array<3xf64>>, index, index, index, !fir.shape<1>) -> !fir.ref<!fir.array<3xf64>>
+ hlfir.assign %el to %wr : !hlfir.expr<3xf64>, !fir.ref<!fir.array<3xf64>>
+ hlfir.destroy %el : !hlfir.expr<3xf64>
+ }
+ return
+}
+
More information about the flang-commits
mailing list