[all-commits] [llvm/llvm-project] fad70c: [ARM] Improve WLS lowering

Thu Mar 11 09:56:48 PST 2021

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: fad70c306854fd21045cad3f6d9226894eb709e6
      https://github.com/llvm/llvm-project/commit/fad70c306854fd21045cad3f6d9226894eb709e6
  Author: David Green <david.green at arm.com>
  Date:   2021-03-11 (Thu, 11 Mar 2021)

  Changed paths:
    M llvm/docs/LangRef.rst
    M llvm/include/llvm/IR/Intrinsics.td
    M llvm/lib/CodeGen/HardwareLoops.cpp
    M llvm/lib/Target/ARM/ARMBaseInstrInfo.cpp
    M llvm/lib/Target/ARM/ARMBaseInstrInfo.h
    M llvm/lib/Target/ARM/ARMBlockPlacement.cpp
    M llvm/lib/Target/ARM/ARMISelDAGToDAG.cpp
    M llvm/lib/Target/ARM/ARMISelLowering.cpp
    M llvm/lib/Target/ARM/ARMISelLowering.h
    M llvm/lib/Target/ARM/ARMInstrThumb2.td
    M llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp
    M llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp
    M llvm/lib/Target/ARM/MVETPAndVPTOptimisationsPass.cpp
    M llvm/lib/Target/ARM/MVETailPredUtils.h
    M llvm/lib/Target/ARM/MVETailPredication.cpp
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/add_reduce.mir
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/biquad-cascade-default.mir
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/biquad-cascade-optsize-strd-lr.mir
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/biquad-cascade-optsize.mir
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/branch-targets.ll
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/fast-fp-loops.ll
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/loop-guards.ll
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/mve-float-loops.ll
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/predicated-liveout.mir
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/revert-non-loop.mir
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/revert-while.mir
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/vmaxmin_vpred_r.mir
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/vmldava_in_vpt.mir
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/while-loops.ll
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/while-negative-offset.mir
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/while.mir
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/wlstp.mir
    M llvm/test/CodeGen/Thumb2/block-placement.mir
    M llvm/test/CodeGen/Thumb2/mve-float16regloops.ll
    M llvm/test/CodeGen/Thumb2/mve-float32regloops.ll
    M llvm/test/CodeGen/Thumb2/mve-postinc-distribute.ll
    M llvm/test/CodeGen/Thumb2/mve-postinc-lsr.ll
    M llvm/test/CodeGen/Thumb2/mve-vmaxnma-commute.ll
    M llvm/test/Transforms/HardwareLoops/ARM/do-rem.ll
    M llvm/test/Transforms/HardwareLoops/ARM/simple-do.ll
    M llvm/test/Transforms/HardwareLoops/ARM/structure.ll
    M llvm/test/Transforms/HardwareLoops/loop-guards.ll
    M llvm/test/Transforms/HardwareLoops/scalar-while.ll

  Log Message:
  -----------
  [ARM] Improve WLS lowering

Recently we improved the lowering of low overhead loops and tail
predicated loops, but concentrated first on the DLS do style loops. This
extends those improvements over to the WLS while loops, improving the
chance of lowering them successfully. To do this the lowering has to
change a little as the instructions are terminators that produce a value
- something that needs to be treated carefully.

Lowering starts at the Hardware Loop pass, inserting a new
llvm.test.start.loop.iterations that produces both an i1 to control the
loop entry and an i32 similar to the llvm.start.loop.iterations
intrinsic added for do loops. This feeds into the loop phi, properly
gluing the values together:

  %wls = call { i32, i1 } @llvm.test.start.loop.iterations.i32(i32 %div)
  %wls0 = extractvalue { i32, i1 } %wls, 0
  %wls1 = extractvalue { i32, i1 } %wls, 1
  br i1 %wls1, label %loop.ph, label %loop.exit
...
loop:
  %lsr.iv = phi i32 [ %wls0, %loop.ph ], [ %iv.next, %loop ]
  ..
  %iv.next = call i32 @llvm.loop.decrement.reg.i32(i32 %lsr.iv, i32 1)
  %cmp = icmp ne i32 %iv.next, 0
  br i1 %cmp, label %loop, label %loop.exit

The llvm.test.start.loop.iterations need to be lowered through ISel
lowering as a pair of WLS and WLSSETUP nodes, which each get converted
to t2WhileLoopSetup and t2WhileLoopStart Pseudos. This helps prevent
t2WhileLoopStart from being a terminator that produces a value,
something difficult to control at that stage in the pipeline. Instead
the t2WhileLoopSetup produces the value of LR (essentially acting as a
lr = subs rn, 0), t2WhileLoopStart consumes that lr value (the Bcc).

These are then converted into a single t2WhileLoopStartLR at the same
point as t2DoLoopStartTP and t2LoopEndDec. Otherwise we revert the loop
to prevent them from progressing further in the pipeline. The
t2WhileLoopStartLR is a single instruction that takes a GPR and produces
LR, similar to the WLS instruction.

  %1:gprlr = t2WhileLoopStartLR %0:rgpr, %bb.3
  t2B %bb.1
...
bb.2.loop:
  %2:gprlr = PHI %1:gprlr, %bb.1, %3:gprlr, %bb.2
  ...
  %3:gprlr = t2LoopEndDec %2:gprlr, %bb.2
  t2B %bb.3

The t2WhileLoopStartLR can then be treated similar to the other low
overhead loop pseudos, eventually being lowered to a WLS providing the
branches are within range.

Differential Revision: https://reviews.llvm.org/D97729

  Commit: bd516d24c1127c98322507a36066fad2083a15e9
      https://github.com/llvm/llvm-project/commit/bd516d24c1127c98322507a36066fad2083a15e9
  Author: David Green <david.green at arm.com>
  Date:   2021-03-11 (Thu, 11 Mar 2021)

  Changed paths:
    M llvm/lib/Target/ARM/ARMISelLowering.cpp
    M llvm/lib/Target/ARM/ARMInstrThumb2.td
    M llvm/lib/Target/ARM/MVETPAndVPTOptimisationsPass.cpp
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/fast-fp-loops.ll
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/loop-guards.ll
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/mve-tail-data-types.ll
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/reductions.ll
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/unpredload.ll
    M llvm/test/CodeGen/Thumb2/LowOverheadLoops/vector-arith-codegen.ll
    M llvm/test/CodeGen/Thumb2/mve-gather-scatter-tailpred.ll
    M llvm/test/CodeGen/Thumb2/mve-postinc-lsr.ll
    M llvm/test/CodeGen/Thumb2/mve-pred-vctpvpsel.ll

  Log Message:
  -----------
  [ARM] Move t2DoLoopStart reg alloc hint

This adjusts the place that the t2DoLoopStart reg allocation hint is
inserted, adding it in the ARMTPAndVPTOptimizaionPass in a similar place
as other tail predicated loop optimizations. This removes the need for
doing so in a custom inserter, and should make the hint more accurate,
only adding it where we expect to create a DLS (not DLSTP or WLS).

Compare: https://github.com/llvm/llvm-project/compare/b68bae6a94fb...bd516d24c112