[llvm] r212714 - [x86, SDAG] Introduce any- and sign-extend-vector-inreg nodes analogous
Chandler Carruth
chandlerc at gmail.com
Thu Jul 10 05:48:21 PDT 2014
FYI, in case someone is interested, it would be good to just mark all of
these as legal for most types when we have SSE4.1 or higher, and add
tablegen patterns to match them directly into (v)pmov{s,z}x instructions.
Currently we're generating shifts blindly, and there is no reason to jump
through all the hoops of the shuffle machinery on SSE4.1 where we have
dedicated instructions to do this...
On Thu, Jul 10, 2014 at 5:32 AM, Chandler Carruth <chandlerc at gmail.com>
wrote:
> Author: chandlerc
> Date: Thu Jul 10 07:32:32 2014
> New Revision: 212714
>
> URL: http://llvm.org/viewvc/llvm-project?rev=212714&view=rev
> Log:
> [x86,SDAG] Introduce any- and sign-extend-vector-inreg nodes analogous
> to the zero-extend-vector-inreg node introduced previously for the same
> purpose: manage the type legalization of widened extend operations,
> especially to support the experimental widening mode for x86.
>
> I'm adding both because sign-extend is expanded in terms of any-extend
> with shifts to propagate the sign bit. This removes the last
> fundamental scalarization from vec_cast2.ll (a test case that hit many
> really bad edge cases for widening legalization), although the trunc
> tests in that file still appear scalarized because the the shuffle
> legalization is scalarizing. Funny thing, I've been working on that.
>
> Some initial experiments with this and SSE2 scenarios is showing
> moderately good behavior already for sign extension. Still some work to
> do on the shuffle combining on X86 before we're generating optimal
> sequences, but avoiding scalarization is a huge step forward.
>
> Modified:
> llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h
> llvm/trunk/include/llvm/CodeGen/SelectionDAG.h
> llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeTypes.h
> llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp
> llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
> llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
> llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp
> llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp
> llvm/trunk/test/CodeGen/X86/vec_cast2.ll
>
> Modified: llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h?rev=212714&r1=212713&r2=212714&view=diff
>
> ==============================================================================
> --- llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h (original)
> +++ llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h Thu Jul 10 07:32:32 2014
> @@ -379,6 +379,28 @@ namespace ISD {
> /// operand, a ValueType node.
> SIGN_EXTEND_INREG,
>
> + /// ANY_EXTEND_VECTOR_INREG(Vector) - This operator represents an
> + /// in-register any-extension of the low lanes of an integer vector.
> The
> + /// result type must have fewer elements than the operand type, and
> those
> + /// elements must be larger integer types such that the total size of
> the
> + /// operand type and the result type match. Each of the low operand
> + /// elements is any-extended into the corresponding, wider result
> + /// elements with the high bits becoming undef.
> + ANY_EXTEND_VECTOR_INREG,
> +
> + /// SIGN_EXTEND_VECTOR_INREG(Vector) - This operator represents an
> + /// in-register sign-extension of the low lanes of an integer vector.
> The
> + /// result type must have fewer elements than the operand type, and
> those
> + /// elements must be larger integer types such that the total size of
> the
> + /// operand type and the result type match. Each of the low operand
> + /// elements is sign-extended into the corresponding, wider result
> + /// elements.
> + // FIXME: The SIGN_EXTEND_INREG node isn't specifically limited to
> + // scalars, but it also doesn't handle vectors well. Either it should
> be
> + // restricted to scalars or this node (and its handling) should be
> merged
> + // into it.
> + SIGN_EXTEND_VECTOR_INREG,
> +
> /// ZERO_EXTEND_VECTOR_INREG(Vector) - This operator represents an
> /// in-register zero-extension of the low lanes of an integer vector.
> The
> /// result type must have fewer elements than the operand type, and
> those
>
> Modified: llvm/trunk/include/llvm/CodeGen/SelectionDAG.h
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/SelectionDAG.h?rev=212714&r1=212713&r2=212714&view=diff
>
> ==============================================================================
> --- llvm/trunk/include/llvm/CodeGen/SelectionDAG.h (original)
> +++ llvm/trunk/include/llvm/CodeGen/SelectionDAG.h Thu Jul 10 07:32:32 2014
> @@ -562,6 +562,18 @@ public:
> /// value assuming it was the smaller SrcTy value.
> SDValue getZeroExtendInReg(SDValue Op, SDLoc DL, EVT SrcTy);
>
> + /// getAnyExtendVectorInReg - Return an operation which will any-extend
> the
> + /// low lanes of the operand into the specified vector type. For
> example,
> + /// this can convert a v16i8 into a v4i32 by any-extending the low four
> + /// lanes of the operand from i8 to i32.
> + SDValue getAnyExtendVectorInReg(SDValue Op, SDLoc DL, EVT VT);
> +
> + /// getSignExtendVectorInReg - Return an operation which will sign
> extend the
> + /// low lanes of the operand into the specified vector type. For
> example,
> + /// this can convert a v16i8 into a v4i32 by sign extending the low four
> + /// lanes of the operand from i8 to i32.
> + SDValue getSignExtendVectorInReg(SDValue Op, SDLoc DL, EVT VT);
> +
> /// getZeroExtendVectorInReg - Return an operation which will zero
> extend the
> /// low lanes of the operand into the specified vector type. For
> example,
> /// this can convert a v16i8 into a v4i32 by zero extending the low four
>
> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeTypes.h
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeTypes.h?rev=212714&r1=212713&r2=212714&view=diff
>
> ==============================================================================
> --- llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeTypes.h (original)
> +++ llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeTypes.h Thu Jul 10
> 07:32:32 2014
> @@ -645,11 +645,11 @@ private:
> bool WidenVectorOperand(SDNode *N, unsigned OpNo);
> SDValue WidenVecOp_BITCAST(SDNode *N);
> SDValue WidenVecOp_CONCAT_VECTORS(SDNode *N);
> + SDValue WidenVecOp_EXTEND(SDNode *N);
> SDValue WidenVecOp_EXTRACT_VECTOR_ELT(SDNode *N);
> SDValue WidenVecOp_EXTRACT_SUBVECTOR(SDNode *N);
> SDValue WidenVecOp_STORE(SDNode* N);
> SDValue WidenVecOp_SETCC(SDNode* N);
> - SDValue WidenVecOp_ZERO_EXTEND(SDNode *N);
>
> SDValue WidenVecOp_Convert(SDNode *N);
>
>
> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp?rev=212714&r1=212713&r2=212714&view=diff
>
> ==============================================================================
> --- llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp (original)
> +++ llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp Thu Jul 10
> 07:32:32 2014
> @@ -75,6 +75,20 @@ class VectorLegalizer {
> /// \brief Implement expansion for SIGN_EXTEND_INREG using SRL and SRA.
> SDValue ExpandSEXTINREG(SDValue Op);
>
> + /// \brief Implement expansion for ANY_EXTEND_VECTOR_INREG.
> + ///
> + /// Shuffles the low lanes of the operand into place and bitcasts to
> the proper
> + /// type. The contents of the bits in the extended part of each element
> are
> + /// undef.
> + SDValue ExpandANY_EXTEND_VECTOR_INREG(SDValue Op);
> +
> + /// \brief Implement expansion for SIGN_EXTEND_VECTOR_INREG.
> + ///
> + /// Shuffles the low lanes of the operand into place, bitcasts to the
> proper
> + /// type, then shifts left and arithmetic shifts right to introduce a
> sign
> + /// extension.
> + SDValue ExpandSIGN_EXTEND_VECTOR_INREG(SDValue Op);
> +
> /// \brief Implement expansion for ZERO_EXTEND_VECTOR_INREG.
> ///
> /// Shuffles the low lanes of the operand into place and blends zeros
> into
> @@ -280,6 +294,8 @@ SDValue VectorLegalizer::LegalizeOp(SDVa
> case ISD::FP_EXTEND:
> case ISD::FMA:
> case ISD::SIGN_EXTEND_INREG:
> + case ISD::ANY_EXTEND_VECTOR_INREG:
> + case ISD::SIGN_EXTEND_VECTOR_INREG:
> case ISD::ZERO_EXTEND_VECTOR_INREG:
> QueryType = Node->getValueType(0);
> break;
> @@ -621,6 +637,10 @@ SDValue VectorLegalizer::Expand(SDValue
> switch (Op->getOpcode()) {
> case ISD::SIGN_EXTEND_INREG:
> return ExpandSEXTINREG(Op);
> + case ISD::ANY_EXTEND_VECTOR_INREG:
> + return ExpandANY_EXTEND_VECTOR_INREG(Op);
> + case ISD::SIGN_EXTEND_VECTOR_INREG:
> + return ExpandSIGN_EXTEND_VECTOR_INREG(Op);
> case ISD::ZERO_EXTEND_VECTOR_INREG:
> return ExpandZERO_EXTEND_VECTOR_INREG(Op);
> case ISD::BSWAP:
> @@ -717,6 +737,52 @@ SDValue VectorLegalizer::ExpandSEXTINREG
> return DAG.getNode(ISD::SRA, DL, VT, Op, ShiftSz);
> }
>
> +// Generically expand a vector anyext in register to a shuffle of the
> relevant
> +// lanes into the appropriate locations, with other lanes left undef.
> +SDValue VectorLegalizer::ExpandANY_EXTEND_VECTOR_INREG(SDValue Op) {
> + SDLoc DL(Op);
> + EVT VT = Op.getValueType();
> + int NumElements = VT.getVectorNumElements();
> + SDValue Src = Op.getOperand(0);
> + EVT SrcVT = Src.getValueType();
> + int NumSrcElements = SrcVT.getVectorNumElements();
> +
> + // Build a base mask of undef shuffles.
> + SmallVector<int, 16> ShuffleMask;
> + ShuffleMask.resize(NumSrcElements, -1);
> +
> + // Place the extended lanes into the correct locations.
> + int ExtLaneScale = NumSrcElements / NumElements;
> + int EndianOffset = TLI.isBigEndian() ? ExtLaneScale - 1 : 0;
> + for (int i = 0; i < NumElements; ++i)
> + ShuffleMask[i * ExtLaneScale + EndianOffset] = i;
> +
> + return DAG.getNode(
> + ISD::BITCAST, DL, VT,
> + DAG.getVectorShuffle(SrcVT, DL, Src, DAG.getUNDEF(SrcVT),
> ShuffleMask));
> +}
> +
> +SDValue VectorLegalizer::ExpandSIGN_EXTEND_VECTOR_INREG(SDValue Op) {
> + SDLoc DL(Op);
> + EVT VT = Op.getValueType();
> + SDValue Src = Op.getOperand(0);
> + EVT SrcVT = Src.getValueType();
> +
> + // First build an any-extend node which can be legalized above when we
> + // recurse through it.
> + Op = DAG.getAnyExtendVectorInReg(Src, DL, VT);
> +
> + // Now we need sign extend. Do this by shifting the elements. Even if
> these
> + // aren't legal operations, they have a better chance of being legalized
> + // without full scalarization than the sign extension does.
> + unsigned EltWidth = VT.getVectorElementType().getSizeInBits();
> + unsigned SrcEltWidth = SrcVT.getVectorElementType().getSizeInBits();
> + SDValue ShiftAmount = DAG.getConstant(EltWidth - SrcEltWidth, VT);
> + return DAG.getNode(ISD::SRA, DL, VT,
> + DAG.getNode(ISD::SHL, DL, VT, Op, ShiftAmount),
> + ShiftAmount);
> +}
> +
> // Generically expand a vector zext in register to a shuffle of the
> relevant
> // lanes into the appropriate locations, a blend of zero into the high
> bits,
> // and a bitcast to the wider element type.
>
> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp?rev=212714&r1=212713&r2=212714&view=diff
>
> ==============================================================================
> --- llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp (original)
> +++ llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp Thu Jul 10
> 07:32:32 2014
> @@ -2398,7 +2398,12 @@ bool DAGTypeLegalizer::WidenVectorOperan
> case ISD::EXTRACT_VECTOR_ELT: Res = WidenVecOp_EXTRACT_VECTOR_ELT(N);
> break;
> case ISD::STORE: Res = WidenVecOp_STORE(N); break;
> case ISD::SETCC: Res = WidenVecOp_SETCC(N); break;
> - case ISD::ZERO_EXTEND: Res = WidenVecOp_ZERO_EXTEND(N); break;
> +
> + case ISD::ANY_EXTEND:
> + case ISD::SIGN_EXTEND:
> + case ISD::ZERO_EXTEND:
> + Res = WidenVecOp_EXTEND(N);
> + break;
>
> case ISD::FP_EXTEND:
> case ISD::FP_TO_SINT:
> @@ -2406,8 +2411,6 @@ bool DAGTypeLegalizer::WidenVectorOperan
> case ISD::SINT_TO_FP:
> case ISD::UINT_TO_FP:
> case ISD::TRUNCATE:
> - case ISD::SIGN_EXTEND:
> - case ISD::ANY_EXTEND:
> Res = WidenVecOp_Convert(N);
> break;
> }
> @@ -2428,14 +2431,14 @@ bool DAGTypeLegalizer::WidenVectorOperan
> return false;
> }
>
> -SDValue DAGTypeLegalizer::WidenVecOp_ZERO_EXTEND(SDNode *N) {
> +SDValue DAGTypeLegalizer::WidenVecOp_EXTEND(SDNode *N) {
> SDLoc DL(N);
> EVT VT = N->getValueType(0);
>
> SDValue InOp = N->getOperand(0);
> // If some legalization strategy other than widening is used on the
> operand,
> - // we can't safely assume that just zero-extending the low lanes is the
> - // correct transformation.
> + // we can't safely assume that just extending the low lanes is the
> correct
> + // transformation.
> if (getTypeAction(InOp.getValueType()) !=
> TargetLowering::TypeWidenVector)
> return WidenVecOp_Convert(N);
> InOp = GetWidenedVector(InOp);
> @@ -2476,9 +2479,18 @@ SDValue DAGTypeLegalizer::WidenVecOp_ZER
> return WidenVecOp_Convert(N);
> }
>
> - // Use a special DAG node to represent the operation of zero extending
> the
> + // Use special DAG nodes to represent the operation of extending the
> // low lanes.
> - return DAG.getZeroExtendVectorInReg(InOp, DL, VT);
> + switch (N->getOpcode()) {
> + default:
> + llvm_unreachable("Extend legalization on on extend operation!");
> + case ISD::ANY_EXTEND:
> + return DAG.getAnyExtendVectorInReg(InOp, DL, VT);
> + case ISD::SIGN_EXTEND:
> + return DAG.getSignExtendVectorInReg(InOp, DL, VT);
> + case ISD::ZERO_EXTEND:
> + return DAG.getZeroExtendVectorInReg(InOp, DL, VT);
> + }
> }
>
> SDValue DAGTypeLegalizer::WidenVecOp_Convert(SDNode *N) {
>
> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp?rev=212714&r1=212713&r2=212714&view=diff
>
> ==============================================================================
> --- llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp (original)
> +++ llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp Thu Jul 10
> 07:32:32 2014
> @@ -1033,6 +1033,26 @@ SDValue SelectionDAG::getZeroExtendInReg
> getConstant(Imm, Op.getValueType()));
> }
>
> +SDValue SelectionDAG::getAnyExtendVectorInReg(SDValue Op, SDLoc DL, EVT
> VT) {
> + assert(VT.isVector() && "This DAG node is restricted to vector types.");
> + assert(VT.getSizeInBits() == Op.getValueType().getSizeInBits() &&
> + "The sizes of the input and result must match in order to
> perform the "
> + "extend in-register.");
> + assert(VT.getVectorNumElements() <
> Op.getValueType().getVectorNumElements() &&
> + "The destination vector type must have fewer lanes than the
> input.");
> + return getNode(ISD::ANY_EXTEND_VECTOR_INREG, DL, VT, Op);
> +}
> +
> +SDValue SelectionDAG::getSignExtendVectorInReg(SDValue Op, SDLoc DL, EVT
> VT) {
> + assert(VT.isVector() && "This DAG node is restricted to vector types.");
> + assert(VT.getSizeInBits() == Op.getValueType().getSizeInBits() &&
> + "The sizes of the input and result must match in order to
> perform the "
> + "extend in-register.");
> + assert(VT.getVectorNumElements() <
> Op.getValueType().getVectorNumElements() &&
> + "The destination vector type must have fewer lanes than the
> input.");
> + return getNode(ISD::SIGN_EXTEND_VECTOR_INREG, DL, VT, Op);
> +}
> +
> SDValue SelectionDAG::getZeroExtendVectorInReg(SDValue Op, SDLoc DL, EVT
> VT) {
> assert(VT.isVector() && "This DAG node is restricted to vector types.");
> assert(VT.getSizeInBits() == Op.getValueType().getSizeInBits() &&
>
> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp?rev=212714&r1=212713&r2=212714&view=diff
>
> ==============================================================================
> --- llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp (original)
> +++ llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp Thu Jul 10
> 07:32:32 2014
> @@ -221,6 +221,8 @@ std::string SDNode::getOperationName(con
> case ISD::ZERO_EXTEND: return "zero_extend";
> case ISD::ANY_EXTEND: return "any_extend";
> case ISD::SIGN_EXTEND_INREG: return "sign_extend_inreg";
> + case ISD::ANY_EXTEND_VECTOR_INREG: return "any_extend_vector_inreg";
> + case ISD::SIGN_EXTEND_VECTOR_INREG: return "sign_extend_vector_inreg";
> case ISD::ZERO_EXTEND_VECTOR_INREG: return "zero_extend_vector_inreg";
> case ISD::TRUNCATE: return "truncate";
> case ISD::FP_ROUND: return "fp_round";
>
> Modified: llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp?rev=212714&r1=212713&r2=212714&view=diff
>
> ==============================================================================
> --- llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp (original)
> +++ llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp Thu Jul 10 07:32:32 2014
> @@ -746,6 +746,10 @@ void TargetLoweringBase::initActions() {
> if (VT >= MVT::FIRST_VECTOR_VALUETYPE &&
> VT <= MVT::LAST_VECTOR_VALUETYPE) {
> setOperationAction(ISD::FCOPYSIGN, (MVT::SimpleValueType)VT,
> Expand);
> + setOperationAction(ISD::ANY_EXTEND_VECTOR_INREG,
> + (MVT::SimpleValueType)VT, Expand);
> + setOperationAction(ISD::SIGN_EXTEND_VECTOR_INREG,
> + (MVT::SimpleValueType)VT, Expand);
> setOperationAction(ISD::ZERO_EXTEND_VECTOR_INREG,
> (MVT::SimpleValueType)VT, Expand);
> }
>
> Modified: llvm/trunk/test/CodeGen/X86/vec_cast2.ll
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/CodeGen/X86/vec_cast2.ll?rev=212714&r1=212713&r2=212714&view=diff
>
> ==============================================================================
> --- llvm/trunk/test/CodeGen/X86/vec_cast2.ll (original)
> +++ llvm/trunk/test/CodeGen/X86/vec_cast2.ll Thu Jul 10 07:32:32 2014
> @@ -4,6 +4,17 @@
> ;CHECK-LABEL: foo1_8:
> ;CHECK: vcvtdq2ps
> ;CHECK: ret
> +;
> +;CHECK-WIDE-LABEL: foo1_8:
> +;CHECK-WIDE: vpmovzxbd %xmm0, %xmm1
> +;CHECK-WIDE-NEXT: vpslld $24, %xmm1, %xmm1
> +;CHECK-WIDE-NEXT: vpsrad $24, %xmm1, %xmm1
> +;CHECK-WIDE-NEXT: vpshufb {{.*}}, %xmm0, %xmm0
> +;CHECK-WIDE-NEXT: vpslld $24, %xmm0, %xmm0
> +;CHECK-WIDE-NEXT: vpsrad $24, %xmm0, %xmm0
> +;CHECK-WIDE-NEXT: vinsertf128 $1, %xmm0, %ymm1, %ymm0
> +;CHECK-WIDE-NEXT: vcvtdq2ps %ymm0, %ymm0
> +;CHECK-WIDE-NEXT: ret
> define <8 x float> @foo1_8(<8 x i8> %src) {
> %res = sitofp <8 x i8> %src to <8 x float>
> ret <8 x float> %res
> @@ -12,6 +23,13 @@ define <8 x float> @foo1_8(<8 x i8> %src
> ;CHECK-LABEL: foo1_4:
> ;CHECK: vcvtdq2ps
> ;CHECK: ret
> +;
> +;CHECK-WIDE-LABEL: foo1_4:
> +;CHECK-WIDE: vpmovzxbd %xmm0, %xmm0
> +;CHECK-WIDE-NEXT: vpslld $24, %xmm0, %xmm0
> +;CHECK-WIDE-NEXT: vpsrad $24, %xmm0, %xmm0
> +;CHECK-WIDE-NEXT: vcvtdq2ps %xmm0, %xmm0
> +;CHECK-WIDE-NEXT: ret
> define <4 x float> @foo1_4(<4 x i8> %src) {
> %res = sitofp <4 x i8> %src to <4 x float>
> ret <4 x float> %res
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20140710/8d059b90/attachment.html>
More information about the llvm-commits
mailing list