[llvm] [SelectionDAG] Avoid unrolling vector scmp/ucmp when widening a narrow result (PR #228568)
via llvm-commits
llvm-commits at lists.llvm.org
Fri Oct 2 13:03:49 PDT 2026
llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-backend-x86
@llvm/pr-subscribers-llvm-selectiondag
Author: Konstantin Bogdanov (thevar1able)
<details>
<summary>Changes</summary>
When a vector `scmp`/`ucmp` result gets widened to a different element count than its operands, `WidenVecRes_CMP` used to fall back to `UnrollVectorOp`. For example, a `<4 x i8>` result widens to `<16 x i8>` on x86 while its `<4 x i32>` operands are already legal.
InstCombine now produces this pattern from `trunc((x >> (BW-1)) | zext(x != 0))` (#<!-- -->196828, #<!-- -->196847). On x86 it gets scalarized into `sets`/`setg`/`pinsrb` chains. x86 has no `ISD::SCMP` lowering, and the integer-promotion change in #<!-- -->198554 (reverted in #<!-- -->204978) doesn't help here.
This patch instead does the comparison in the operand type, then sign-extends or truncates to the result element type, which keeps the values `-1/0/1`. If the target expands the comparison, it's expanded right away so the truncate is combined with the setccs and x86 can use `PACKSS`.
```
scmp <4 x i8> (<4 x i32> %x, zeroinitializer), -mcpu=x86-64-v3
before: 25 instructions (vpextrd/testl/sets/setg/subb/vpinsrb x4)
after: vpxor, vpcmpgtd, vpcmpgtd, vpsubd, vpshufb
```
Reproducer (trunk, without this patch): https://godbolt.org/z/dfEaPsYvr
```c
void sign(const int *restrict x, signed char *restrict out, int n) {
for (int i = 0; i < n; i++)
out[i] = (x[i] >> 31) | (x[i] != 0);
}
```
The vector loop body goes from 57 to 19 instructions with this patch.
It also adds a `TargetLowering::shouldUnrollVectorCMP` hook so targets can keep unrolling where vector compares are too expensive. X86 uses it for vXi64 before SSE4.2, where an i64 compare is emulated with several vXi32 compares and shuffles. llvm-mca throughput with SSE2 only, in cycles:
| | scalar | vector |
|---|---|---|
| `scmp <16 x i64> → <16 x i8>` (x86-64 / skylake / znver3) | 38.3 / 39.0 / 32.0 | 39.5 / 47.0 / 25.3 |
| `scmp <2 x i64> → <2 x i8>` (x86-64 / skylake / znver3) | 5.0 / 4.0 / 5.0 | 6.0 / 8.0 / 4.0 |
With the hook in place, no X86 test gets worse. Some examples:
| test | before | after |
|---|---|---|
| `scmp_wide_vec_op` (SSE4 / AVX2 / AVX512) | 114 / 123 / 127 | 50 / 24 / 16 |
| `ucmp_wide_vec_op` (SSE2 / SSE4 / AVX2) | 115 / 169 / 103 | 29 / 31 / 18 |
(Counts are lines in the CHECK output.)
Non-simple operand types (e.g. `<17 x i71>`), scalable vectors, and i1 operands keep the existing unrolling.
The first commit adds the new tests with the current codegen (NFC); the second contains the change and its test diff.
---
Patch is 66.80 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/228568.diff
6 Files Affected:
- (modified) llvm/include/llvm/CodeGen/TargetLowering.h (+6)
- (modified) llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp (+13-1)
- (modified) llvm/lib/Target/X86/X86ISelLowering.cpp (+4)
- (modified) llvm/lib/Target/X86/X86ISelLowering.h (+4)
- (modified) llvm/test/CodeGen/X86/scmp.ll (+425-620)
- (modified) llvm/test/CodeGen/X86/ucmp.ll (+108-473)
``````````diff
diff --git a/llvm/include/llvm/CodeGen/TargetLowering.h b/llvm/include/llvm/CodeGen/TargetLowering.h
index 46521fa8c544a..f2e7bc88f9116 100644
--- a/llvm/include/llvm/CodeGen/TargetLowering.h
+++ b/llvm/include/llvm/CodeGen/TargetLowering.h
@@ -3692,6 +3692,12 @@ class LLVM_ABI TargetLoweringBase {
return isOperationLegalOrCustom(Op, VT);
}
+ /// Should a vector SCMP/UCMP with operands of type OpVT be unrolled
+ /// rather than computed in the operand type, when its result type needs to
+ /// be widened to a different element count. Return true if vector
+ /// comparisons of OpVT are expensive enough that scalar code is faster.
+ virtual bool shouldUnrollVectorCMP(EVT OpVT) const { return false; }
+
/// Should we prefer selects to doing arithmetic on boolean types
virtual bool preferSelectsOverBooleanArithmetic(EVT VT) const {
return false;
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
index ef88647a16dee..31937bb322c0b 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
@@ -5680,7 +5680,19 @@ SDValue DAGTypeLegalizer::WidenVecRes_CMP(SDNode *N) {
return DAG.getNode(N->getOpcode(), dl, WidenResVT, LHS, RHS);
}
- return DAG.UnrollVectorOp(N, WidenResVT.getVectorNumElements());
+ if (!OpVT.isSimple() || OpVT.isScalableVector() ||
+ OpVT.getScalarSizeInBits() < 2 || TLI.shouldUnrollVectorCMP(OpVT))
+ return DAG.UnrollVectorOp(N, WidenResVT.getVectorNumElements());
+
+ SDValue Cmp = DAG.getNode(N->getOpcode(), dl, OpVT, LHS, RHS);
+ if (TLI.getOperationAction(N->getOpcode(), OpVT) == TargetLowering::Expand)
+ Cmp = TLI.expandCMP(Cmp.getNode(), DAG);
+ EVT ResEltVT = WidenResVT.getVectorElementType();
+ Cmp =
+ DAG.getSExtOrTrunc(Cmp, dl, OpVT.changeVectorElementType(Ctxt, ResEltVT));
+ if (OpVT.getVectorNumElements() < WidenResVT.getVectorNumElements())
+ return DAG.getInsertSubvector(dl, DAG.getUNDEF(WidenResVT), Cmp, 0);
+ return DAG.getExtractSubvector(dl, WidenResVT, Cmp, 0);
}
SDValue DAGTypeLegalizer::WidenVecRes_BinaryWithExtraScalarOp(SDNode *N) {
diff --git a/llvm/lib/Target/X86/X86ISelLowering.cpp b/llvm/lib/Target/X86/X86ISelLowering.cpp
index a84d017d9c6cd..3bd1d701591d1 100644
--- a/llvm/lib/Target/X86/X86ISelLowering.cpp
+++ b/llvm/lib/Target/X86/X86ISelLowering.cpp
@@ -3699,6 +3699,10 @@ X86TargetLowering::getExtractSubvectorCost(EVT ResVT, EVT SrcVT,
return ExtractSubvectorCost::Expensive;
}
+bool X86TargetLowering::shouldUnrollVectorCMP(EVT OpVT) const {
+ return OpVT.getScalarType() == MVT::i64 && !Subtarget.hasSSE42();
+}
+
bool X86TargetLowering::shouldScalarizeBinop(SDValue VecOp) const {
unsigned Opc = VecOp.getOpcode();
diff --git a/llvm/lib/Target/X86/X86ISelLowering.h b/llvm/lib/Target/X86/X86ISelLowering.h
index ba04080573c42..c72c160a05217 100644
--- a/llvm/lib/Target/X86/X86ISelLowering.h
+++ b/llvm/lib/Target/X86/X86ISelLowering.h
@@ -590,6 +590,10 @@ namespace llvm {
/// supported.
bool shouldScalarizeBinop(SDValue) const override;
+ /// Pre-SSE42 vXi64 comparisons have to be emulated with multiple vXi32
+ /// comparisons and shuffles, which is slower than scalar code.
+ bool shouldUnrollVectorCMP(EVT OpVT) const override;
+
/// Extract of a scalar FP value from index 0 of a vector is free.
bool isExtractVecEltCheap(EVT VT, unsigned Index) const override {
EVT EltVT = VT.getScalarType();
diff --git a/llvm/test/CodeGen/X86/scmp.ll b/llvm/test/CodeGen/X86/scmp.ll
index 393e05bfd0cc6..f6893fc439ded 100644
--- a/llvm/test/CodeGen/X86/scmp.ll
+++ b/llvm/test/CodeGen/X86/scmp.ll
@@ -777,119 +777,34 @@ define <4 x i32> @scmp_normal_vectors(<4 x i32> %x, <4 x i32> %y) nounwind {
}
define <4 x i8> @scmp_narrow_vec_result(<4 x i32> %x, <4 x i32> %y) nounwind {
-; SSE2-LABEL: scmp_narrow_vec_result:
-; SSE2: # %bb.0:
-; SSE2-NEXT: movd %xmm1, %eax
-; SSE2-NEXT: movd %xmm0, %ecx
-; SSE2-NEXT: cmpl %eax, %ecx
-; SSE2-NEXT: setl %al
-; SSE2-NEXT: setg %cl
-; SSE2-NEXT: subb %al, %cl
-; SSE2-NEXT: movzbl %cl, %eax
-; SSE2-NEXT: pshufd {{.*#+}} xmm2 = xmm1[1,1,1,1]
-; SSE2-NEXT: movd %xmm2, %ecx
-; SSE2-NEXT: pshufd {{.*#+}} xmm2 = xmm0[1,1,1,1]
-; SSE2-NEXT: movd %xmm2, %edx
-; SSE2-NEXT: cmpl %ecx, %edx
-; SSE2-NEXT: setl %cl
-; SSE2-NEXT: setg %dl
-; SSE2-NEXT: subb %cl, %dl
-; SSE2-NEXT: movzbl %dl, %ecx
-; SSE2-NEXT: shll $8, %ecx
-; SSE2-NEXT: orl %eax, %ecx
-; SSE2-NEXT: pshufd {{.*#+}} xmm2 = xmm1[2,3,2,3]
-; SSE2-NEXT: movd %xmm2, %eax
-; SSE2-NEXT: pshufd {{.*#+}} xmm2 = xmm0[2,3,2,3]
-; SSE2-NEXT: movd %xmm2, %edx
-; SSE2-NEXT: cmpl %eax, %edx
-; SSE2-NEXT: setl %al
-; SSE2-NEXT: setg %dl
-; SSE2-NEXT: subb %al, %dl
-; SSE2-NEXT: movzbl %dl, %eax
-; SSE2-NEXT: shll $16, %eax
-; SSE2-NEXT: orl %ecx, %eax
-; SSE2-NEXT: pshufd {{.*#+}} xmm1 = xmm1[3,3,3,3]
-; SSE2-NEXT: movd %xmm1, %ecx
-; SSE2-NEXT: pshufd {{.*#+}} xmm0 = xmm0[3,3,3,3]
-; SSE2-NEXT: movd %xmm0, %edx
-; SSE2-NEXT: cmpl %ecx, %edx
-; SSE2-NEXT: setl %cl
-; SSE2-NEXT: setg %dl
-; SSE2-NEXT: subb %cl, %dl
-; SSE2-NEXT: movzbl %dl, %ecx
-; SSE2-NEXT: shll $24, %ecx
-; SSE2-NEXT: orl %eax, %ecx
-; SSE2-NEXT: movd %ecx, %xmm0
-; SSE2-NEXT: retq
+; SSE-LABEL: scmp_narrow_vec_result:
+; SSE: # %bb.0:
+; SSE-NEXT: movdqa %xmm0, %xmm2
+; SSE-NEXT: pcmpgtd %xmm1, %xmm2
+; SSE-NEXT: pcmpgtd %xmm0, %xmm1
+; SSE-NEXT: psubd %xmm2, %xmm1
+; SSE-NEXT: packssdw %xmm1, %xmm1
+; SSE-NEXT: packsswb %xmm1, %xmm1
+; SSE-NEXT: movdqa %xmm1, %xmm0
+; SSE-NEXT: retq
;
-; SSE4-LABEL: scmp_narrow_vec_result:
-; SSE4: # %bb.0:
-; SSE4-NEXT: pextrd $1, %xmm1, %eax
-; SSE4-NEXT: pextrd $1, %xmm0, %ecx
-; SSE4-NEXT: cmpl %eax, %ecx
-; SSE4-NEXT: setl %al
-; SSE4-NEXT: setg %cl
-; SSE4-NEXT: subb %al, %cl
-; SSE4-NEXT: movzbl %cl, %eax
-; SSE4-NEXT: movd %xmm1, %ecx
-; SSE4-NEXT: movd %xmm0, %edx
-; SSE4-NEXT: cmpl %ecx, %edx
-; SSE4-NEXT: setl %cl
-; SSE4-NEXT: setg %dl
-; SSE4-NEXT: subb %cl, %dl
-; SSE4-NEXT: movzbl %dl, %ecx
-; SSE4-NEXT: movd %ecx, %xmm2
-; SSE4-NEXT: pinsrb $1, %eax, %xmm2
-; SSE4-NEXT: pextrd $2, %xmm1, %eax
-; SSE4-NEXT: pextrd $2, %xmm0, %ecx
-; SSE4-NEXT: cmpl %eax, %ecx
-; SSE4-NEXT: setl %al
-; SSE4-NEXT: setg %cl
-; SSE4-NEXT: subb %al, %cl
-; SSE4-NEXT: movzbl %cl, %eax
-; SSE4-NEXT: pinsrb $2, %eax, %xmm2
-; SSE4-NEXT: pextrd $3, %xmm1, %eax
-; SSE4-NEXT: pextrd $3, %xmm0, %ecx
-; SSE4-NEXT: cmpl %eax, %ecx
-; SSE4-NEXT: setl %al
-; SSE4-NEXT: setg %cl
-; SSE4-NEXT: subb %al, %cl
-; SSE4-NEXT: movzbl %cl, %eax
-; SSE4-NEXT: pinsrb $3, %eax, %xmm2
-; SSE4-NEXT: movdqa %xmm2, %xmm0
-; SSE4-NEXT: retq
+; AVX2-LABEL: scmp_narrow_vec_result:
+; AVX2: # %bb.0:
+; AVX2-NEXT: vpcmpgtd %xmm1, %xmm0, %xmm2
+; AVX2-NEXT: vpcmpgtd %xmm0, %xmm1, %xmm0
+; AVX2-NEXT: vpsubd %xmm2, %xmm0, %xmm0
+; AVX2-NEXT: vpshufb {{.*#+}} xmm0 = xmm0[0,4,8,12,0,4,8,12,0,4,8,12,0,4,8,12]
+; AVX2-NEXT: retq
;
-; AVX-LABEL: scmp_narrow_vec_result:
-; AVX: # %bb.0:
-; AVX-NEXT: vpextrd $1, %xmm1, %eax
-; AVX-NEXT: vpextrd $1, %xmm0, %ecx
-; AVX-NEXT: cmpl %eax, %ecx
-; AVX-NEXT: setl %al
-; AVX-NEXT: setg %cl
-; AVX-NEXT: subb %al, %cl
-; AVX-NEXT: vmovd %xmm1, %eax
-; AVX-NEXT: vmovd %xmm0, %edx
-; AVX-NEXT: cmpl %eax, %edx
-; AVX-NEXT: setl %al
-; AVX-NEXT: setg %dl
-; AVX-NEXT: subb %al, %dl
-; AVX-NEXT: vmovd %edx, %xmm2
-; AVX-NEXT: vpinsrb $1, %ecx, %xmm2, %xmm2
-; AVX-NEXT: vpextrd $2, %xmm1, %eax
-; AVX-NEXT: vpextrd $2, %xmm0, %ecx
-; AVX-NEXT: cmpl %eax, %ecx
-; AVX-NEXT: setl %al
-; AVX-NEXT: setg %cl
-; AVX-NEXT: subb %al, %cl
-; AVX-NEXT: vpinsrb $2, %ecx, %xmm2, %xmm2
-; AVX-NEXT: vpextrd $3, %xmm1, %eax
-; AVX-NEXT: vpextrd $3, %xmm0, %ecx
-; AVX-NEXT: cmpl %eax, %ecx
-; AVX-NEXT: setl %al
-; AVX-NEXT: setg %cl
-; AVX-NEXT: subb %al, %cl
-; AVX-NEXT: vpinsrb $3, %ecx, %xmm2, %xmm0
-; AVX-NEXT: retq
+; AVX512-LABEL: scmp_narrow_vec_result:
+; AVX512: # %bb.0:
+; AVX512-NEXT: vpcmpgtd %xmm0, %xmm1, %k1
+; AVX512-NEXT: vpcmpgtd %xmm1, %xmm0, %k2
+; AVX512-NEXT: vpbroadcastd {{.*#+}} xmm0 {%k2} {z} = [1,1,1,1]
+; AVX512-NEXT: vpcmpeqd %xmm1, %xmm1, %xmm1
+; AVX512-NEXT: vmovdqa32 %xmm1, %xmm0 {%k1}
+; AVX512-NEXT: vpmovdb %xmm0, %xmm0
+; AVX512-NEXT: retq
;
; X86-LABEL: scmp_narrow_vec_result:
; X86: # %bb.0:
@@ -925,98 +840,311 @@ define <4 x i8> @scmp_narrow_vec_result(<4 x i32> %x, <4 x i32> %y) nounwind {
; X86-NEXT: popl %edi
; X86-NEXT: popl %ebx
; X86-NEXT: retl $4
+ %1 = call <4 x i8> @llvm.scmp(<4 x i32> %x, <4 x i32> %y)
+ ret <4 x i8> %1
+}
+
+; scmp(x, 0) with a narrow result, as formed by InstCombine from
+; trunc((x >> (BW-1)) | zext(x != 0)).
+define <4 x i8> @scmp_zero_narrow_vec_result(<4 x i32> %x) nounwind {
+; SSE-LABEL: scmp_zero_narrow_vec_result:
+; SSE: # %bb.0:
+; SSE-NEXT: pxor %xmm1, %xmm1
+; SSE-NEXT: movdqa %xmm0, %xmm2
+; SSE-NEXT: pcmpgtd %xmm1, %xmm2
+; SSE-NEXT: pcmpgtd %xmm0, %xmm1
+; SSE-NEXT: psubd %xmm2, %xmm1
+; SSE-NEXT: packssdw %xmm1, %xmm1
+; SSE-NEXT: packsswb %xmm1, %xmm1
+; SSE-NEXT: movdqa %xmm1, %xmm0
+; SSE-NEXT: retq
+;
+; AVX2-LABEL: scmp_zero_narrow_vec_result:
+; AVX2: # %bb.0:
+; AVX2-NEXT: vpxor %xmm1, %xmm1, %xmm1
+; AVX2-NEXT: vpcmpgtd %xmm1, %xmm0, %xmm2
+; AVX2-NEXT: vpcmpgtd %xmm0, %xmm1, %xmm0
+; AVX2-NEXT: vpsubd %xmm2, %xmm0, %xmm0
+; AVX2-NEXT: vpshufb {{.*#+}} xmm0 = xmm0[0,4,8,12,0,4,8,12,0,4,8,12,0,4,8,12]
+; AVX2-NEXT: retq
+;
+; AVX512-LABEL: scmp_zero_narrow_vec_result:
+; AVX512: # %bb.0:
+; AVX512-NEXT: vpxor %xmm1, %xmm1, %xmm1
+; AVX512-NEXT: vpcmpgtd %xmm1, %xmm0, %k1
+; AVX512-NEXT: vpsrad $31, %xmm0, %xmm0
+; AVX512-NEXT: vpord {{\.?LCPI[0-9]+_[0-9]+}}(%rip){1to4}, %xmm0, %xmm0 {%k1}
+; AVX512-NEXT: vpmovdb %xmm0, %xmm0
+; AVX512-NEXT: retq
+;
+; X86-LABEL: scmp_zero_narrow_vec_result:
+; X86: # %bb.0:
+; X86-NEXT: pushl %ebx
+; X86-NEXT: movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT: cmpl $0, {{[0-9]+}}(%esp)
+; X86-NEXT: sets %dl
+; X86-NEXT: setg %cl
+; X86-NEXT: subb %dl, %cl
+; X86-NEXT: cmpl $0, {{[0-9]+}}(%esp)
+; X86-NEXT: sets %dl
+; X86-NEXT: setg %ch
+; X86-NEXT: subb %dl, %ch
+; X86-NEXT: cmpl $0, {{[0-9]+}}(%esp)
+; X86-NEXT: sets %dl
+; X86-NEXT: setg %dh
+; X86-NEXT: subb %dl, %dh
+; X86-NEXT: cmpl $0, {{[0-9]+}}(%esp)
+; X86-NEXT: sets %dl
+; X86-NEXT: setg %bl
+; X86-NEXT: subb %dl, %bl
+; X86-NEXT: movb %bl, 3(%eax)
+; X86-NEXT: movb %dh, 2(%eax)
+; X86-NEXT: movb %ch, 1(%eax)
+; X86-NEXT: movb %cl, (%eax)
+; X86-NEXT: popl %ebx
+; X86-NEXT: retl $4
+ %1 = call <4 x i8> @llvm.scmp(<4 x i32> %x, <4 x i32> zeroinitializer)
+ ret <4 x i8> %1
+}
+
+define <8 x i8> @scmp_narrow_vec_result_v8i16(<8 x i16> %x, <8 x i16> %y) nounwind {
+; SSE-LABEL: scmp_narrow_vec_result_v8i16:
+; SSE: # %bb.0:
+; SSE-NEXT: movdqa %xmm0, %xmm2
+; SSE-NEXT: pcmpgtw %xmm1, %xmm2
+; SSE-NEXT: pcmpgtw %xmm0, %xmm1
+; SSE-NEXT: psubw %xmm2, %xmm1
+; SSE-NEXT: packsswb %xmm1, %xmm1
+; SSE-NEXT: movdqa %xmm1, %xmm0
+; SSE-NEXT: retq
+;
+; AVX2-LABEL: scmp_narrow_vec_result_v8i16:
+; AVX2: # %bb.0:
+; AVX2-NEXT: vpcmpgtw %xmm1, %xmm0, %xmm2
+; AVX2-NEXT: vpcmpgtw %xmm0, %xmm1, %xmm0
+; AVX2-NEXT: vpsubw %xmm2, %xmm0, %xmm0
+; AVX2-NEXT: vpacksswb %xmm0, %xmm0, %xmm0
+; AVX2-NEXT: retq
+;
+; AVX512-LABEL: scmp_narrow_vec_result_v8i16:
+; AVX512: # %bb.0:
+; AVX512-NEXT: vpcmpgtw %xmm0, %xmm1, %k1
+; AVX512-NEXT: vpcmpgtw %xmm1, %xmm0, %k2
+; AVX512-NEXT: vmovdqu16 {{.*#+}} xmm0 {%k2} {z} = [1,1,1,1,1,1,1,1]
+; AVX512-NEXT: vpcmpeqd %xmm1, %xmm1, %xmm1
+; AVX512-NEXT: vmovdqu16 %xmm1, %xmm0 {%k1}
+; AVX512-NEXT: vpacksswb %xmm0, %xmm0, %xmm0
+; AVX512-NEXT: retq
+;
+; X86-LABEL: scmp_narrow_vec_result_v8i16:
+; X86: # %bb.0:
+; X86-NEXT: pushl %ebp
+; X86-NEXT: pushl %ebx
+; X86-NEXT: pushl %edi
+; X86-NEXT: pushl %esi
+; X86-NEXT: pushl %eax
+; X86-NEXT: movzwl {{[0-9]+}}(%esp), %ebp
+; X86-NEXT: movzwl {{[0-9]+}}(%esp), %esi
+; X86-NEXT: movzwl {{[0-9]+}}(%esp), %edi
+; X86-NEXT: movzwl {{[0-9]+}}(%esp), %edx
+; X86-NEXT: movzwl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT: cmpw {{[0-9]+}}(%esp), %cx
+; X86-NEXT: setl %ch
+; X86-NEXT: setg %al
+; X86-NEXT: subb %ch, %al
+; X86-NEXT: movb %al, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Spill
+; X86-NEXT: cmpw {{[0-9]+}}(%esp), %dx
+; X86-NEXT: setl %ch
+; X86-NEXT: setg %al
+; X86-NEXT: subb %ch, %al
+; X86-NEXT: movb %al, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Spill
+; X86-NEXT: movzwl {{[0-9]+}}(%esp), %ebx
+; X86-NEXT: cmpw {{[0-9]+}}(%esp), %bx
+; X86-NEXT: setl %dh
+; X86-NEXT: setg %ch
+; X86-NEXT: subb %dh, %ch
+; X86-NEXT: cmpw {{[0-9]+}}(%esp), %di
+; X86-NEXT: setl %bl
+; X86-NEXT: setg %dh
+; X86-NEXT: subb %bl, %dh
+; X86-NEXT: movzwl {{[0-9]+}}(%esp), %edi
+; X86-NEXT: cmpw {{[0-9]+}}(%esp), %di
+; X86-NEXT: setl %bh
+; X86-NEXT: setg %bl
+; X86-NEXT: subb %bh, %bl
+; X86-NEXT: cmpw {{[0-9]+}}(%esp), %si
+; X86-NEXT: setl %al
+; X86-NEXT: setg %bh
+; X86-NEXT: subb %al, %bh
+; X86-NEXT: movzwl {{[0-9]+}}(%esp), %eax
+; X86-NEXT: cmpw {{[0-9]+}}(%esp), %ax
+; X86-NEXT: setl %al
+; X86-NEXT: setg %dl
+; X86-NEXT: subb %al, %dl
+; X86-NEXT: cmpw {{[0-9]+}}(%esp), %bp
+; X86-NEXT: setl %al
+; X86-NEXT: setg %cl
+; X86-NEXT: subb %al, %cl
+; X86-NEXT: movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT: movb %cl, 7(%eax)
+; X86-NEXT: movb %dl, 6(%eax)
+; X86-NEXT: movb %bh, 5(%eax)
+; X86-NEXT: movb %bl, 4(%eax)
+; X86-NEXT: movb %dh, 3(%eax)
+; X86-NEXT: movb %ch, 2(%eax)
+; X86-NEXT: movzbl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 1-byte Folded Reload
+; X86-NEXT: movb %cl, 1(%eax)
+; X86-NEXT: movzbl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 1-byte Folded Reload
+; X86-NEXT: movb %cl, (%eax)
+; X86-NEXT: addl $4, %esp
+; X86-NEXT: popl %esi
+; X86-NEXT: popl %edi
+; X86-NEXT: popl %ebx
+; X86-NEXT: popl %ebp
+; X86-NEXT: retl $4
+ %1 = call <8 x i8> @llvm.scmp(<8 x i16> %x, <8 x i16> %y)
+ ret <8 x i8> %1
+}
+
+define <2 x i8> @scmp_narrow_vec_result_v2i64(<2 x i64> %x, <2 x i64> %y) nounwind {
+; SSE2-LABEL: scmp_narrow_vec_result_v2i64:
+; SSE2: # %bb.0:
+; SSE2-NEXT: movq %xmm1, %rax
+; SSE2-NEXT: movq %xmm0, %rcx
+; SSE2-NEXT: cmpq %rax, %rcx
+; SSE2-NEXT: setl %al
+; SSE2-NEXT: setg %cl
+; SSE2-NEXT: subb %al, %cl
+; SSE2-NEXT: movzbl %cl, %eax
+; SSE2-NEXT: pshufd {{.*#+}} xmm1 = xmm1[2,3,2,3]
+; SSE2-NEXT: movq %xmm1, %rcx
+; SSE2-NEXT: pshufd {{.*#+}} xmm0 = xmm0[2,3,2,3]
+; SSE2-NEXT: movq %xmm0, %rdx
+; SSE2-NEXT: cmpq %rcx, %rdx
+; SSE2-NEXT: setl %cl
+; SSE2-NEXT: setg %dl
+; SSE2-NEXT: subb %cl, %dl
+; SSE2-NEXT: movzbl %dl, %ecx
+; SSE2-NEXT: shll $8, %ecx
+; SSE2-NEXT: orl %eax, %ecx
+; SSE2-NEXT: movd %ecx, %xmm0
+; SSE2-NEXT: retq
+;
+; SSE4-LABEL: scmp_narrow_vec_result_v2i64:
+; SSE4: # %bb.0:
+; SSE4-NEXT: movdqa %xmm0, %xmm2
+; SSE4-NEXT: pcmpgtq %xmm1, %xmm2
+; SSE4-NEXT: pcmpgtq %xmm0, %xmm1
+; SSE4-NEXT: psubq %xmm2, %xmm1
+; SSE4-NEXT: packssdw %xmm1, %xmm1
+; SSE4-NEXT: packssdw %xmm1, %xmm1
+; SSE4-NEXT: packsswb %xmm1, %xmm1
+; SSE4-NEXT: movdqa %xmm1, %xmm0
+; SSE4-NEXT: retq
;
-; SETZUCC-LABEL: scmp_narrow_vec_result:
+; AVX2-LABEL: scmp_narrow_vec_result_v2i64:
+; AVX2: # %bb.0:
+; AVX2-NEXT: vpcmpgtq %xmm1, %xmm0, %xmm2
+; AVX2-NEXT: vpcmpgtq %xmm0, %xmm1, %xmm0
+; AVX2-NEXT: vpsubq %xmm2, %xmm0, %xmm0
+; AVX2-NEXT: vpackssdw %xmm0, %xmm0, %xmm0
+; AVX2-NEXT: vpackssdw %xmm0, %xmm0, %xmm0
+; AVX2-NEXT: vpacksswb %xmm0, %xmm0, %xmm0
+; AVX2-NEXT: retq
+;
+; AVX512-LABEL: scmp_narrow_vec_result_v2i64:
+; AVX512: # %bb.0:
+; AVX512-NEXT: vpcmpgtq %xmm0, %xmm1, %k1
+; AVX512-NEXT: vpcmpgtq %xmm1, %xmm0, %k2
+; AVX512-NEXT: vpbroadcastq {{.*#+}} xmm0 {%k2} {z} = [1,1]
+; AVX512-NEXT: vpcmpeqd %xmm1, %xmm1, %xmm1
+; AVX512-NEXT: vmovdqa64 %xmm1, %xmm0 {%k1}
+; AVX512-NEXT: vpmovqb %xmm0, %xmm0
+; AVX512-NEXT: retq
+;
+; X86-LABEL: scmp_narrow_vec_result_v2i64:
+; X86: # %bb.0:
+; X86-NEXT: pushl %ebp
+; X86-NEXT: pushl %ebx
+; X86-NEXT: pushl %edi
+; X86-NEXT: pushl %esi
+; X86-NEXT: movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT: movl {{[0-9]+}}(%esp), %edx
+; X86-NEXT: movl {{[0-9]+}}(%esp), %esi
+; X86-NEXT: movl {{[0-9]+}}(%esp), %edi
+; X86-NEXT: cmpl %eax, %esi
+; X86-NEXT: movl %edi, %ecx
+; X86-NEXT: sbbl %edx, %ecx
+; X86-NEXT: movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT: setl %bl
+; X86-NEXT: cmpl %esi, %eax
+; X86-NEXT: movl {{[0-9]+}}(%esp), %esi
+; X86-NEXT: movl {{[0-9]+}}(%esp), %ebp
+; X86-NEXT: sbbl %edi, %edx
+; X86-NEXT: movl {{[0-9]+}}(%esp), %edx
+; X86-NEXT: setl %al
+; X86-NEXT: subb %bl, %al
+; X86-NEXT: cmpl %ecx, %ebp
+; X86-NEXT: movl %edx, %edi
+; X86-NEXT: sbbl %esi, %edi
+; X86-NEXT: setl %ah
+; X86-NEXT: cmpl %ebp, %ecx
+; X86-NEXT: sbbl %edx, %esi
+; X86-NEXT: setl %dl
+; X86-NEXT: subb %ah, %dl
+; X86-NEXT: popl %esi
+; X86-NEXT: popl %edi
+; X86-NEXT: popl %ebx
+; X86-NEXT: popl %ebp
+; X86-NEXT: retl
+;
+; SETZUCC-LABEL: scmp_narrow_vec_result_v2i64:
; SETZUCC: # %bb.0:
-; SETZUCC-NEXT: movd %xmm1, %eax
-; SETZUCC-NEXT: movd %xmm0, %ecx
-; SETZUCC-NEXT: cmpl %eax, %ecx
+; SETZUCC-NEXT: movq %xmm1, %rax
+; SETZUCC-NEXT: movq %xmm0, %rcx
+; SETZUCC-NEXT: cmpq %rax, %rcx
; SETZUCC-NEXT: setzul %al
; SETZUCC-NEXT: setzug %cl
; SETZUCC-NEXT: subb %al, %cl
; SETZUCC-NEXT: movzbl %cl, %eax
-; SETZUCC-NEXT: pshufd {{.*#+}} xmm2 = xmm1[1,1,1,1]
-; SETZUCC-NEXT: movd %xmm2, %ecx
-; SETZUCC-NEXT: pshufd {{.*#+}} xmm2 = xmm0[1,1,1,1]
-; SETZUCC-NEXT: movd %xmm2, %edx
-; SETZUCC-NEXT: cmpl %ecx, %edx
+; SETZUCC-NEXT: pshufd {{.*#+}} xmm1 = xmm1[2,3,2,3]
+; SETZUCC-NEXT: movq %xmm1, %rcx
+; SETZUCC-NEXT: pshufd {{.*#+}} xmm0 = xmm0[2,3,2,3]
+; SETZUCC-NEXT: movq %xmm0, %rdx
+; SETZUCC-NEXT: cmpq %rcx, %rdx
; SETZUCC-NEXT: setzul %cl
; SETZUCC-NEXT: setzug %dl
; SETZUCC-NEXT: subb %cl, %dl
; SETZUCC-NEXT: movzbl %dl, %ecx
; SETZUCC-NEXT: shll $8, %ecx
; SETZUCC-NEXT: orl %eax, %ecx
-; SETZUCC-NEXT: pshufd {{.*#+}} xmm2 = xmm1[2,3,2,3]
-; SETZUCC-NEXT: movd %xmm2, %eax
-; SETZUCC-NEXT: pshufd {{.*#+}} xmm2 = xmm0[2,3,2,3]
-; SETZUCC-NEXT: movd %xmm2, %edx
-; SETZUCC-NEXT: cmpl %eax, %edx
-; SETZUCC-NEXT: setzul %al
-; SETZUCC-NEXT: setzug %dl
-; SETZUCC-NEXT: subb %al, %dl
-; SETZUCC-NEXT: movzbl %dl, %eax
-; SETZUCC-NEXT: shll $16, %eax
-; SETZUCC-NEXT: orl %ecx, %eax
-; SETZUCC-NEXT: pshufd {{.*#+}} xmm1 = xmm1[3,3,3,3]
-; SETZUCC-NEXT: movd %xmm1, %ecx
-; SETZUCC-NEXT: pshufd {{.*#+}} xmm0 = xmm0[3,3,3,3]
-; SETZUCC-NEXT: movd %xmm0, %edx
-; SETZUCC-NEXT: cmpl %ecx, %edx
-; SETZUCC-NEXT: setzul %cl
-; SETZUCC-NEXT: setzug %dl
-; SETZUCC-NEXT: subb %cl, %dl
-; SETZUCC-NEXT: movzbl %dl, %ecx
-; SETZUCC-NEXT: shll $24, %ecx
-; SETZUCC-NEXT: orl %eax, %ecx
; SETZUCC-NEXT: movd %ecx, %xmm0
; SETZUCC-NEXT: retq
;
-; NO-SETZUCC-LABEL: scmp_narrow_vec_result:
+; NO-SETZUCC-LABEL: scmp_narrow_vec_result_v2i64:
; NO-SETZUCC: # %bb.0:
-; NO-SETZUCC-NEXT: movd %xmm1, %eax
-; NO-SETZUCC-NEXT: movd %xmm0, %ecx
-; NO-SETZUCC-NEXT: cmpl %eax, %ecx
+; NO-SETZUCC-NEXT: movq %xmm1, %rax
+; NO-SETZUCC-NEXT: movq %xmm0, %rcx
+; NO-SETZUCC-NEXT: cmpq %rax, %rcx
; NO-SETZUCC-NEXT: setl %al
; NO-SETZUCC-NEXT: setg %cl
; NO-SETZUCC-NEXT: subb %al, %cl
; NO-SETZUCC-NEXT: movzbl %cl, %eax
-; NO-SETZUCC-NEXT: pshufd {{.*#+}} xmm2 = xmm1[1,1,1,1]
-; NO-SETZUCC-NEXT: movd %xmm2, %ecx
-; NO-SETZUCC-NEXT: pshufd {{.*#+}} xmm2 = xmm0[1,1,1,1]
-; NO-SETZUCC-NEXT:...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/228568
More information about the llvm-commits
mailing list