[llvm] [LV] Handle FSub Partial Reductions (PR #191186)
Benjamin Maxwell via llvm-commits
llvm-commits at lists.llvm.org
Thu Apr 16 03:33:34 PDT 2026
================
@@ -9104,17 +9058,30 @@ static SmallVector<Instruction *> preparePlanForEpilogueVectorLoop(
// Partial sub-reductions always start at 0 and account for the
// reduction start value in a final subtraction. Update it to use the
// resume value from the main vector loop.
- if (PhiR->getVFScaleFactor() > 1 &&
- PhiR->getRecurrenceKind() == RecurKind::Sub) {
+ if ((PhiR->getVFScaleFactor() > 1 &&
+ PhiR->getRecurrenceKind() == RecurKind::Sub) ||
+ PhiR->getRecurrenceKind() == RecurKind::FSub) {
auto *Sub = cast<VPInstruction>(RdxResult->getSingleUser());
- assert(Sub->getOpcode() == Instruction::Sub && "Unexpected opcode");
+ assert((Sub->getOpcode() == Instruction::Sub ||
+ Sub->getOpcode() == Instruction::FSub) &&
+ "Unexpected opcode");
assert(isa<VPIRValue>(Sub->getOperand(0)) &&
"Expected operand to match the original start value of the "
"reduction");
- assert(VPlanPatternMatch::match(VPI->getOperand(0),
- VPlanPatternMatch::m_ZeroInt()) &&
- "Expected start value for partial sub-reduction to start at "
- "zero");
+ // For integer sub-reductions, verify start value is zero.
+ // For FP sub-reductions, verify start value is negative zero.
+ if (PhiR->getRecurrenceKind() == RecurKind::Sub) {
+ assert(VPlanPatternMatch::match(VPI->getOperand(0),
+ VPlanPatternMatch::m_ZeroInt()) &&
+ "Expected start value for partial sub-reduction to start "
+ "at zero");
+ } else if (PhiR->getRecurrenceKind() == RecurKind::FSub) {
+ assert(
+ VPlanPatternMatch::match(VPI->getOperand(0),
+ VPlanPatternMatch::m_NegZeroFP()) &&
+ "Expected start value for partial sub-reduction to start "
+ "at negative zero");
+ }
----------------
MacDue wrote:
nit: I think you write this as:
```c++
// For integer sub-reductions, verify start value is zero.
// For FP sub-reductions, verify start value is negative zero.
[[maybe_unused]] auto IsStartValueIsIdentity = [&] {
Value *IdentityValue = getRecurrenceIdentity(
PhiR->getRecurrenceKind(), ResumeV->getType(), {});
auto *StartValue = dyn_cast<VPIRValue>(VPI->getOperand(0));
return StartValue && StartValue->getValue() == IdentityValue;
};
assert(IsStartValueIsIdentity() &&
"Expected start value for partial sub-reduction to be the "
"zero (or negative zero)");
```
Which would help simplify this patch as we don't need `m_NegZeroFP()` (some code paths of that matcher seem untested).
https://github.com/llvm/llvm-project/pull/191186
More information about the llvm-commits
mailing list