[llvm] [AMDGPU] Filter candidates of LiveRegOptimizer for profitable cases (PR #124624)

[Jeffrey Byrnes via llvm-commits]

================
@@ -125,8 +130,41 @@ class LiveRegOptimizer {
     return LK.first != TargetLoweringBase::TypeLegal;
   }
 
-  LiveRegOptimizer(Module &Mod, const GCNSubtarget &ST)
-      : Mod(Mod), DL(Mod.getDataLayout()), ST(ST),
+  // Filtering based on operation or its cost.
+  // If an operation incurs high enough cost or natively work on
+  // vector of illegal type, ie. v2i8, then it makes sense to try
+  // to avoid scalarizing across BB.
+  bool shouldReplaceBasedOnOp(Instruction *II) {
+    // Ignore pseudos
+    if (II->isDebugOrPseudoInst())
+      return false;
+
+    // Instruction Cost
+    const auto Cost = TTI.getInstructionCost(
+        II, TargetTransformInfo::TargetCostKind::TCK_SizeAndLatency);
+    LLVM_DEBUG(dbgs() << "shouldReplaceBasedOnOp: " << *II << " Cost=" << Cost
+                      << '\n';);
+    if (Cost >= 8)
+      return true;
+
+    // Intrinsics - assume they natively handle illegal type
+    if (dyn_cast<IntrinsicInst>(II))
+      return true;
+
+    // Stores
+    if (dyn_cast<StoreInst>(II))
+      return true;
+
+    // Shuffles
+    if (dyn_cast<ShuffleVectorInst>(II))
----------------
jrbyrnes wrote:

I think the most accurate heuristic would be akin to what other passes (e.g. SLP) does, in that we calculate the aggregate value of vectorizing (coercing the illegal type) vs scalarizing (keeping the illegal type as is). But I think that we probably don't need that level of modeling.

For the cases where we have mixed uses (e.g. some user instructions which can natively operate on i8 vectors, and some user instructions which must be scalarized), I would think it is best to prefer type coercion. When we introduce the scalarization in block, I think we have a better chance of folding the vector extract code into the scalarized ops themselves (i.e. sdwa). Moreover, in the case of loops, the livein vector will be live throughout the body of the loop. The coercion will reduce the loop's baseline RP. 

https://github.com/llvm/llvm-project/pull/124624