[PATCH] D149281: Must unroll epilogue loops after vectorization on AMDGPU target

[Alexander via Phabricator via llvm-commits]

alex-t created this revision.
alex-t added reviewers: rampitec, fhahn.
Herald added subscribers: kosarev, foad, StephenFan, kerbowa, hiraditya, tpr, dstuttard, yaxunl, jvesely, kzhuravl, arsenm.
Herald added a project: All.
alex-t requested review of this revision.
Herald added subscribers: pcwang-thead, wdng.
Herald added a project: LLVM.

We've got a performance regression after the https://reviews.llvm.org/D115261.
Despite the loop being vectorized unroll is still required.


https://reviews.llvm.org/D149281

Files:
  llvm/include/llvm/Analysis/TargetTransformInfo.h
  llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
  llvm/lib/Transforms/Vectorize/LoopVectorize.cpp


Index: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
===================================================================
--- llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -7778,7 +7778,10 @@
     LoopVectorizeHints Hints(L, true, *ORE);
     Hints.setAlreadyVectorized();
   }
-  AddRuntimeUnrollDisableMetaData(L);
+  TargetTransformInfo::UnrollingPreferences UP;
+  TTI->getUnrollingPreferences(L, *PSE.getSE(), UP, ORE);
+  if (!UP.unrollLoopEpilogues)
+    AddRuntimeUnrollDisableMetaData(L);
 
   // 3. Fix the vectorized code: take care of header phi's, live-outs,
   //    predication, updating analyses.
Index: llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
===================================================================
--- llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
+++ llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
@@ -113,6 +113,9 @@
   // manipulations in average.
   UP.BEInsns += 3;
 
+  // We want to run unroll for epilog loops.
+  UP.unrollLoopEpilogues = true;
+
   // TODO: Do we want runtime unrolling?
 
   // Maximum alloca size than can fit registers. Reserve 16 registers.
Index: llvm/include/llvm/Analysis/TargetTransformInfo.h
===================================================================
--- llvm/include/llvm/Analysis/TargetTransformInfo.h
+++ llvm/include/llvm/Analysis/TargetTransformInfo.h
@@ -573,6 +573,8 @@
     /// Don't allow loop unrolling to simulate more than this number of
     /// iterations when checking full unroll profitability
     unsigned MaxIterationsCountToAnalyze;
+    /// Enable runtime unrolling when vectorizing the epilogue loop. 
+    bool unrollLoopEpilogues = false;
   };
 
   /// Get target-customized preferences for the generic loop unrolling


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D149281.517259.patch
Type: text/x-patch
Size: 1801 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20230426/eead6afe/attachment.bin>