[all-commits] [llvm/llvm-project] 8114ef: [LoopVectorize] Support vectorization of compressi...

[Benjamin Maxwell via All-commits]

  Branch: refs/heads/users/MacDue/compressed.vec
  Home:   https://github.com/llvm/llvm-project
  Commit: 8114ef51cc53a7b77d3cd2e61984f607b0908d65
      https://github.com/llvm/llvm-project/commit/8114ef51cc53a7b77d3cd2e61984f607b0908d65
  Author: Sergey Kachkov <sergey.kachkov at syntacore.com>
  Date:   2026-09-23 (Wed, 23 Sep 2026)

  Changed paths:
    M llvm/include/llvm/Analysis/VectorUtils.h
    M llvm/include/llvm/Transforms/Utils/LoopUtils.h
    M llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
    M llvm/lib/Analysis/VectorUtils.cpp
    M llvm/lib/IR/IntrinsicInst.cpp
    M llvm/lib/Transforms/Utils/LoopUtils.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
    M llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
    M llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h
    M llvm/lib/Transforms/Vectorize/VPlan.cpp
    M llvm/lib/Transforms/Vectorize/VPlan.h
    M llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
    M llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
    M llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
    M llvm/lib/Transforms/Vectorize/VPlanTransforms.h
    M llvm/lib/Transforms/Vectorize/VPlanUtils.cpp
    A llvm/test/Transforms/LoopVectorize/VPlan/compress-idioms.ll
    M llvm/test/Transforms/LoopVectorize/VPlan/vplan-print-before-after-all.ll
    M llvm/test/Transforms/LoopVectorize/compress-idioms-negative-tests.ll
    M llvm/unittests/Transforms/Vectorize/VPlanTestBase.h

  Log Message:
  -----------
  [LoopVectorize] Support vectorization of compressing patterns in VPlan

RFC link: https://discourse.llvm.org/t/rfc-loop-vectorization-of-compress-store-expand-load-patterns/86442

This adds loop vectorizer support for "compressing" patterns,
for example:

```
int dst_idx = 0;
for (int i = 0; i < n; i++) {
  if (cond[i])
    dst[dst_idx++] = src[i];
}
```

Can be vectorized with a `llvm.masked.compressstore` as:

```
int dst_idx = 0;
for (int i = 0; i < n; i++) {
  %cond = load(%cond) != 0
  %src = masked.load(%src[i], %cond)
  masked.compressstore(%src, %dst[dst_idx], %cond)
  dst_idx += num.active.lanes(%cond)
}
```

and:

```
int src_idx = 0;
for (int i = 0; i < n; i++) {
  if (cond[i])
    dst[i] = src[src_idx++];
}
```

Can be vectorized with a `llvm.masked.expandload` as:

```
int src_idx = 0;
for (int i = 0; i < n; i++) {
  %cond = load(%cond) != 0
  %src = masked.expandload(%src[src_idx], %cond)
  masked.store(%src, %dst[i], %cond)
  src_idx += num.active.lanes(%cond)
}
```

This uses the new `MonotonicDescriptor` to recognize
monotonic/compressing patterns. The phis are mapped to a new
`VPMonotonicPHIRecipe`, this will map to a scalar phi. We only
allow uniform uses of monotonic phis in the loop (e.g., as the pointer
to a compressed load/store).

Compressed loads/stores are recognized with
`LoopVectorizationLegality::isCompressedPtr`. Currently, we only allow
cases where:

- The (monotonic) pointer has a stride equal to the access size
- The memory operation is predicated with the same condition as the increment

This is a continuation Sergey Kachkov's patch (#140723).

There are a number of changes from the initial patch:

- Expandloads/compresstores directly use `VPWidenMemIntrinsic`
- ComputeMonotonicResult is replaced with existing VP instructions
- AArch64, VPlan, and target-agnostic tests have been added
- This style of vectorization if off by default
  - The switch can be flipped soon after this patch lands

Tests, fixes, and design rework

Add comment

Add out-of-loop use check

Fixups

Fixups

Fixups

Add test

Remove AArch64 flags

Rename MonotonicPHI (and related) to ConditionalInduction*

Add other target tests

Flip flag


  Commit: 8c0fb4b24e37bf960a9af04d93b5968b140dae09
      https://github.com/llvm/llvm-project/commit/8c0fb4b24e37bf960a9af04d93b5968b140dae09
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-23 (Wed, 23 Sep 2026)

  Changed paths:
    M llvm/test/Transforms/LoopVectorize/AArch64/compress-idioms.ll
    M llvm/test/Transforms/LoopVectorize/X86/compress-idioms.ll
    M llvm/test/Transforms/LoopVectorize/compress-idioms.ll
    M llvm/test/Transforms/LoopVectorize/compress-store-vec-epilogue.ll

  Log Message:
  -----------
  Update checks


  Commit: 9b1dbf0a07d348d0b5426425af27812aecc13656
      https://github.com/llvm/llvm-project/commit/9b1dbf0a07d348d0b5426425af27812aecc13656
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-23 (Wed, 23 Sep 2026)

  Changed paths:
    M llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
    M llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

  Log Message:
  -----------
  Always pass a vector type to TTI.isLegalMaskedCompressStore/ExpandLoad


  Commit: cdabf8f5146b80b2d8377e6e0d149ed89721f616
      https://github.com/llvm/llvm-project/commit/cdabf8f5146b80b2d8377e6e0d149ed89721f616
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-23 (Wed, 23 Sep 2026)

  Changed paths:
    M llvm/lib/Transforms/Utils/LoopUtils.cpp

  Log Message:
  -----------
  Rebase fixups


  Commit: d8d6957668d1df06725b70e6e3f38fabf10777c8
      https://github.com/llvm/llvm-project/commit/d8d6957668d1df06725b70e6e3f38fabf10777c8
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-23 (Wed, 23 Sep 2026)

  Changed paths:
    M llvm/include/llvm/Transforms/Utils/LoopUtils.h

  Log Message:
  -----------
  Add LLVM_ABI marker


Compare: https://github.com/llvm/llvm-project/compare/7bd1a4527813...d8d6957668d1

To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications