[all-commits] [llvm/llvm-project] 622e81: [VPlan] Fix VPInstruction::AnyOf combine undoing f...

[Luke Lau via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 622e816cb4bf94ea99e2f1763c5c16b061ec80e2
      https://github.com/llvm/llvm-project/commit/622e816cb4bf94ea99e2f1763c5c16b061ec80e2
  Author: Luke Lau <luke at igalia.com>
  Date:   2026-09-21 (Mon, 21 Sep 2026)

  Changed paths:
    M llvm/lib/Transforms/Vectorize/VPlan.h
    M llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
    M llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
    M llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
    M llvm/test/Transforms/LoopVectorize/AArch64/cmp_cost.ll
    M llvm/test/Transforms/LoopVectorize/AArch64/early_exit_with_stores.ll
    M llvm/test/Transforms/LoopVectorize/AArch64/fmax-without-fast-math-flags.ll
    M llvm/test/Transforms/LoopVectorize/AArch64/fmin-without-fast-math-flags.ll
    M llvm/test/Transforms/LoopVectorize/AArch64/single-early-exit-interleave.ll
    M llvm/test/Transforms/LoopVectorize/AArch64/sve-non-native-interleaved-accesses.ll
    M llvm/test/Transforms/LoopVectorize/RISCV/early_exit_with_stores.ll
    M llvm/test/Transforms/LoopVectorize/VPlan/conditional-scalar-assignment-vplan.ll
    M llvm/test/Transforms/LoopVectorize/VPlan/early_exit_with_stores_vplan.ll
    M llvm/test/Transforms/LoopVectorize/VPlan/interleave-conditional-scalar-assignment-vplan.ll
    M llvm/test/Transforms/LoopVectorize/VPlan/uncountable-early-exit-vplan.ll
    M llvm/test/Transforms/LoopVectorize/X86/cost-any-of.ll
    M llvm/test/Transforms/LoopVectorize/early_exit_with_stores.ll
    M llvm/test/Transforms/LoopVectorize/epilog-vectorization-fmaxnum-reductions.ll
    M llvm/test/Transforms/LoopVectorize/fcmp-uno-fold-interleave.ll
    M llvm/test/Transforms/LoopVectorize/fmax-without-fast-math-flags-interleave.ll
    M llvm/test/Transforms/LoopVectorize/fmax-without-fast-math-flags.ll
    M llvm/test/Transforms/LoopVectorize/fmin-without-fast-math-flags.ll
    M llvm/test/Transforms/LoopVectorize/interleave_uncountable_exits.ll
    M llvm/test/Transforms/LoopVectorize/iv-select-cmp-non-const-iv-start.ll
    M llvm/test/Transforms/LoopVectorize/iv-select-cmp-trunc.ll
    M llvm/test/Transforms/LoopVectorize/iv-select-cmp.ll
    M llvm/test/Transforms/LoopVectorize/minmax_reduction.ll
    M llvm/test/Transforms/LoopVectorize/scalarized_conditional_ops_uncountable_exits.ll
    M llvm/test/Transforms/LoopVectorize/select-cmp.ll
    A llvm/test/Transforms/LoopVectorize/single-early-exit-anyof-fold.ll
    M llvm/test/Transforms/LoopVectorize/single-early-exit-interleave-hint.ll
    M llvm/test/Transforms/LoopVectorize/single-early-exit-interleave.ll
    M llvm/test/Transforms/LoopVectorize/tail_fold_uncountable_exits.ll

  Log Message:
  -----------
  [VPlan] Fix VPInstruction::AnyOf combine undoing freeze (#223640)

There is an any-of combine for unrolled VPlans which does:

any-of (fcmp uno A, A), (fcmp uno B, B), ...-> any-of (fcmp uno A, B)

However any-of implicitly freezes each individual operand and this means
we go from `freeze (fcmp uno A, A)` to `freeze (fcmp uno A, B)` which
isn't sound: https://alive2.llvm.org/ce/z/UdQM7C

This causes miscompiles today with early exit loops that are unrolled,
see the attached test case in single-early-exit-anyof-fold.ll.

This fixes it by explicitly modelling the freeze in VPlan. There are
three places where we use AnyOf:

1) early exit loops: the freeze needs to be applied per-lane, so apply
it as `(any-of (freeze (combined-conds-to-exit)))`
2) handleMaxMinNumReductions: we need to freeze the operands to the
min/max reductions earlier since they have multiple uses introduced and
we don't check the values aren't undef
3) handleFindLastReductions: we need to freeze Cond itself since it's
got multiple uses, but I plan on fixing this in a separate PR

Fable 5.1 was used to help generate the tests



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications