[all-commits] [llvm/llvm-project] 12ce70: [LoopUnroll] Skip remainder loop guard if skip unr...

[Joel E. Denny via All-commits]

  Branch: refs/heads/users/jdenny-ornl/skip-unroll-epilog-guard
  Home:   https://github.com/llvm/llvm-project
  Commit: 12ce70ef97bde190851dc02e1d58711f83bd6c80
      https://github.com/llvm/llvm-project/commit/12ce70ef97bde190851dc02e1d58711f83bd6c80
  Author: Joel E. Denny <jdenny.ornl at gmail.com>
  Date:   2025-09-02 (Tue, 02 Sep 2025)

  Changed paths:
    M llvm/lib/Transforms/Utils/LoopUnrollRuntime.cpp
    M llvm/test/DebugInfo/KeyInstructions/Generic/loop-unroll-runtime.ll
    M llvm/test/Transforms/HardwareLoops/ARM/structure.ll
    M llvm/test/Transforms/LoopUnroll/AArch64/apple-unrolling.ll
    M llvm/test/Transforms/LoopUnroll/AArch64/runtime-unroll-generic.ll
    M llvm/test/Transforms/LoopUnroll/AArch64/vector.ll
    M llvm/test/Transforms/LoopUnroll/AMDGPU/unroll-runtime.ll
    M llvm/test/Transforms/LoopUnroll/ARM/multi-blocks.ll
    M llvm/test/Transforms/LoopUnroll/Hexagon/reuse-lcssa-phi-scev-expansion.ll
    M llvm/test/Transforms/LoopUnroll/PowerPC/p8-unrolling-legalize-vectors-inseltpoison.ll
    M llvm/test/Transforms/LoopUnroll/PowerPC/p8-unrolling-legalize-vectors.ll
    M llvm/test/Transforms/LoopUnroll/RISCV/vector.ll
    M llvm/test/Transforms/LoopUnroll/WebAssembly/basic-unrolling.ll
    M llvm/test/Transforms/LoopUnroll/convergent.controlled.ll
    M llvm/test/Transforms/LoopUnroll/followup.ll
    M llvm/test/Transforms/LoopUnroll/runtime-epilog-debuginfo.ll
    M llvm/test/Transforms/LoopUnroll/runtime-exit-phi-scev-invalidation.ll
    M llvm/test/Transforms/LoopUnroll/runtime-i128.ll
    M llvm/test/Transforms/LoopUnroll/runtime-loop-at-most-two-exits.ll
    M llvm/test/Transforms/LoopUnroll/runtime-loop-branchweight.ll
    M llvm/test/Transforms/LoopUnroll/runtime-loop-multiple-exits.ll
    M llvm/test/Transforms/LoopUnroll/runtime-loop.ll
    M llvm/test/Transforms/LoopUnroll/runtime-loop1.ll
    M llvm/test/Transforms/LoopUnroll/runtime-loop2.ll
    M llvm/test/Transforms/LoopUnroll/runtime-loop5.ll
    M llvm/test/Transforms/LoopUnroll/runtime-multiexit-heuristic.ll
    M llvm/test/Transforms/LoopUnroll/runtime-unroll-assume-no-remainder.ll
    M llvm/test/Transforms/LoopUnroll/runtime-unroll-reductions.ll
    M llvm/test/Transforms/LoopUnroll/runtime-unroll-remainder.ll
    M llvm/test/Transforms/LoopUnroll/scev-invalidation-lcssa.ll
    M llvm/test/Transforms/LoopUnroll/tripcount-overflow.ll
    M llvm/test/Transforms/LoopUnroll/unroll-heuristics-pgo.ll
    M llvm/test/Transforms/LoopUnroll/unroll-loads-cse.ll
    M llvm/test/Transforms/LoopUnrollAndJam/dependencies_visit_order.ll
    M llvm/test/Transforms/LoopUnrollAndJam/followup.ll
    M llvm/test/Transforms/LoopUnrollAndJam/unroll-and-jam.ll
    M llvm/test/Transforms/LoopVectorize/X86/float-induction-x86.ll
    M llvm/test/Transforms/PhaseOrdering/AArch64/extra-unroll-simplifications.ll

  Log Message:
  -----------
  [LoopUnroll] Skip remainder loop guard if skip unrolled loop

The original loop (OL) that serves as input to LoopUnroll has basic
blocks that are arranged as follows:

```
OLPreHeader
OLHeader <-.
...        |
OLLatch ---'
OLExit
```

In this depiction, every block has an implicit edge to the next block
below, so any explicit edge indicates a conditional branch.

Given OL and unroll count N, LoopUnroll sometimes creates an unrolled
loop (UL) with a remainder loop (RL) epilogue arranged like this:

```
,-- ULGuard
|   ULPreHeader
|   ULHeader <-.
|   ...        |
|   ULLatch ---'
|   ULExit
`-> RLGuard -----.
    RLPreHeader  |
,-> RLHeader     |
|   ...          |
`-- RLLatch      |
    RLExit       |
    OLExit <-----'
```

Each UL iteration executes N OL iterations, but each RL iteration
executes 1 OL iteration.  ULGuard or RLGuard checks whether the first
iteration of UL or RL should execute, respectively.  If so, ULLatch or
RLLatch checks whether to execute each subsequent iteration.

Once reached, OL always executes its first iteration but not
necessarily the next N-1 iterations.  Thus, ULGuard is always required
before the first UL iteration.  However, when control flows from
ULGuard directly to RLGuard, the first OL iteration has yet to
execute, so RLGuard is then redundant before the first RL iteration.

Thus, this patch makes the following changes:
- Adjust ULGuard to branch to RLPreHeader instead of RLGuard, thus
  eliminating RLGuard's unnecessary branch instruction for that path.
- Eliminate the creation of RLGuard phi node poison values.  Without
  this patch, RLGuard has such a phi node for each value that is
  defined by any OL iteration and used in OLExit.  The poison value is
  required where ULGuard is the predecessor.  The poison value
  indicates that control flow from ULGuard to RLGuard to Exit has no
  counterpart in OL because the first OL iteration must execute either
  in UL or RL.
- Simplify the CFG by not splitting ULExit and RLGuard because,
  without the ULGuard predecessor, the single block can now be a
  dedicated UL exit.
- To RLPreHeader, add an `llvm.assume` call that asserts the RL trip
  count is non-zero.  Without this patch, RLPreHeader is reachable
  only when RLGuard guarantees that assertion is true.  With this
  patch, RLGuard guarantees it only when RLGuard is the predecessor,
  and the OL structure guarantees it when ULGuard is the predecessor.
  If RL itself is unrolled later, this guarantee somehow prevents
  ScalarEvolution from giving up when trying to compute a maximum trip
  count for RL.  That maximum trip count enables the branch
  instruction in the final unrolled instance of RLLatch to be
  eliminated.  Without the `llvm.assume` call, some existing unroll
  tests start to fail because that instruction is not eliminated.

The original motivation for this patch is to facilitate later patches
that fix LoopUnroll's computation of branch weights so that they
maintain the block frequency of OL's body.  Specifically, this patch
ensures RLGuard's branch weights do not affect RL's contribution to
the block frequency of OL's body in the case that ULGuard skips UL.



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications