[llvm] [SelectionDAG] Improve wide integer squaring (PR #226403)

[MITSUNARI Shigeo via llvm-commits]

https://github.com/herumi created https://github.com/llvm/llvm-project/pull/226403

For a wide integer `mul x, x`, the number of limb multiplications is
n(n+1)/2: n for x[i]*x[i] (i=0, ..., n-1) and n(n-1)/2 for x[i]*x[j]
(i<j). The existing expansion already reaches this count. However, the
way the products are added has room for improvement. The existing
expansion adds each cross product x[i]*x[j] twice and materializes the
carry between partial sums with setcc/zext before adding it back, so the
add and carry-handling instructions outnumber the multiplications
several times over, and register pressure goes up as well.

This patch adds TargetLowering::expandWideSquare. It groups the products
x[i]*x[j] into rows by diagonal (the products with equal j-i) and
accumulates them from the narrowest width up as
`acc = (acc << U) + row`. The products on one diagonal do not
overlap, so a row is built by concatenation alone, adding a row fits in
a single adc chain, and the carries stay within the row. Finally, acc is
doubled (shl 1) and the concatenation of x[i]*x[i] is added. This
removes the materialized carries (setb/movzbl) and minimizes the number
of adc chains.

The limb type is getTypeToExpandTo() of the result type, i.e. the legal
integer type the result is ultimately expanded to (i64 on x86-64,
AArch64 and RISCV64, i32 on i386 and RISCV32). The expansion is used
only when this type has UMUL_LOHI or MULHU, and falls back to the
existing expansion otherwise. The operand is usually zero-extended (to
get all the bits of the square of an i384, it is zero-extended to i768
and then multiplied), so the limbs that computeKnownBits proves to be
zero are left out and the value is squared at its real width. When the
result is truncated (e.g. i512 = mul i512 %x, %x), the row widths are
clamped to the number of result limbs and the square is computed modulo
2^Bits (a product that reaches the top limb is computed with a plain mul
for its low half only).

The result is as follows.

The tables show the number of non-multiply instructions for an input of
n 64-bit limbs (an i(64n) with all limbs in use is zero-extended to
i(128n) and squared to get the full product; n=6 means i384 x i384 ->
i768), i.e. the whole function minus the multiply instructions. The
number of multiplications is the same n(n+1)/2 for org and opti: mulx on
x86-64, and the two instructions mul + umulh on AArch64. The value in
parentheses is the throughput time ratio to org (ns/op, 4 independent
streams; measured on a Xeon w9-3495X for x86-64 and an Apple M4 Pro for
AArch64).

x86-64 (+bmi2)

| n | 6 | 8 | 16 | 32 |
|---|---|---|---|---|
| org | 215 | 390 | 1654 | 6790 |
| opti | 138 (0.74x) | 246 (0.77x) | 1014 (0.77x) | 4037 (0.53x) |

Breakdown for n=6 (both use 21 mulx): org has adc 58 / add 38 /
setb 13 / movzbl 13, opti has adc 30 / add 6 / shld 9 / setb 0.

AArch64 (-mcpu=apple-m1)

| n | 6 | 8 | 16 | 32 |
|---|---|---|---|---|
| org | 145 | 293 | 1434 | 6344 |
| opti | 61 (0.62x) | 98 (0.48x) | 501 (0.49x) | 3117 (0.60x) |

Breakdown for n=6 (both use 21 mul + 21 umulh): org has adds 46 /
adcs 25 / cinc 36, opti has adds 5 / adcs 25 / cinc 4 / extr 9.

Tests: added llc output for `mul iK %x, %x` (including zext / sext
variants) on X86 / AArch64 / RISCV. The only existing test that changes
is the i192 square in X86/dagcombine-cse.ll, which gets shorter.

Assisted-by: Claude Code


>From 578f5f1681f8d452900ddfc29bc8fc3fc0a2c5e3 Mon Sep 17 00:00:00 2001
From: MITSUNARI Shigeo <herumi at nifty.com>
Date: Thu, 17 Sep 2026 11:46:38 +0900
Subject: [PATCH] [SelectionDAG] Improve wide integer squaring

For a wide integer `mul x, x`, the number of limb multiplications is
n(n+1)/2: n for x[i]*x[i] (i=0, ..., n-1) and n(n-1)/2 for x[i]*x[j]
(i<j). The existing expansion already reaches this count. However, the
way the products are added has room for improvement. The existing
expansion adds each cross product x[i]*x[j] twice and materializes the
carry between partial sums with setcc/zext before adding it back, so the
add and carry-handling instructions outnumber the multiplications
several times over, and register pressure goes up as well.

This patch adds TargetLowering::expandWideSquare. It groups the products
x[i]*x[j] into rows by diagonal (the products with equal j-i) and
accumulates them from the narrowest width up as
`acc = (acc << U) + row`. The products on one diagonal do not
overlap, so a row is built by concatenation alone, adding a row fits in
a single adc chain, and the carries stay within the row. Finally, acc is
doubled (shl 1) and the concatenation of x[i]*x[i] is added. This
removes the materialized carries (setb/movzbl) and minimizes the number
of adc chains.

The limb type is getTypeToExpandTo() of the result type, i.e. the legal
integer type the result is ultimately expanded to (i64 on x86-64,
AArch64 and RISCV64, i32 on i386 and RISCV32). The expansion is used
only when this type has UMUL_LOHI or MULHU, and falls back to the
existing expansion otherwise. The operand is usually zero-extended (to
get all the bits of the square of an i384, it is zero-extended to i768
and then multiplied), so the limbs that computeKnownBits proves to be
zero are left out and the value is squared at its real width. When the
result is truncated (e.g. i512 = mul i512 %x, %x), the row widths are
clamped to the number of result limbs and the square is computed modulo
2^Bits (a product that reaches the top limb is computed with a plain mul
for its low half only).

The result is as follows.

The tables show the number of non-multiply instructions for an input of
n 64-bit limbs (an i(64n) with all limbs in use is zero-extended to
i(128n) and squared to get the full product; n=6 means i384 x i384 ->
i768), i.e. the whole function minus the multiply instructions. The
number of multiplications is the same n(n+1)/2 for org and opti: mulx on
x86-64, and the two instructions mul + umulh on AArch64. The value in
parentheses is the throughput time ratio to org (ns/op, 4 independent
streams; measured on a Xeon w9-3495X for x86-64 and an Apple M4 Pro for
AArch64).

x86-64 (+bmi2)

| n | 6 | 8 | 16 | 32 |
|---|---|---|---|---|
| org | 215 | 390 | 1654 | 6790 |
| opti | 138 (0.74x) | 246 (0.77x) | 1014 (0.77x) | 4037 (0.53x) |

Breakdown for n=6 (both use 21 mulx): org has adc 58 / add 38 /
setb 13 / movzbl 13, opti has adc 30 / add 6 / shld 9 / setb 0.

AArch64 (-mcpu=apple-m1)

| n | 6 | 8 | 16 | 32 |
|---|---|---|---|---|
| org | 145 | 293 | 1434 | 6344 |
| opti | 61 (0.62x) | 98 (0.48x) | 501 (0.49x) | 3117 (0.60x) |

Breakdown for n=6 (both use 21 mul + 21 umulh): org has adds 46 /
adcs 25 / cinc 36, opti has adds 5 / adcs 25 / cinc 4 / extr 9.

Tests: added llc output for `mul iK %x, %x` (including zext / sext
variants) on X86 / AArch64 / RISCV. The only existing test that changes
is the i192 square in X86/dagcombine-cse.ll, which gets shorter.
---
 llvm/include/llvm/CodeGen/TargetLowering.h    |   6 +
 .../SelectionDAG/LegalizeIntegerTypes.cpp     |   7 +
 .../CodeGen/SelectionDAG/TargetLowering.cpp   | 135 +++
 llvm/test/CodeGen/AArch64/wide-int-square.ll  | 288 +++++++
 llvm/test/CodeGen/RISCV/wide-int-square.ll    | 723 ++++++++++++++++
 llvm/test/CodeGen/X86/dagcombine-cse.ll       |  93 +-
 llvm/test/CodeGen/X86/wide-int-square-i512.ll | 302 +++++++
 llvm/test/CodeGen/X86/wide-int-square.ll      | 809 ++++++++++++++++++
 8 files changed, 2314 insertions(+), 49 deletions(-)
 create mode 100644 llvm/test/CodeGen/AArch64/wide-int-square.ll
 create mode 100644 llvm/test/CodeGen/RISCV/wide-int-square.ll
 create mode 100644 llvm/test/CodeGen/X86/wide-int-square-i512.ll
 create mode 100644 llvm/test/CodeGen/X86/wide-int-square.ll

diff --git a/llvm/include/llvm/CodeGen/TargetLowering.h b/llvm/include/llvm/CodeGen/TargetLowering.h
index c08d0e53ec34b3..9b56f988877918 100644
--- a/llvm/include/llvm/CodeGen/TargetLowering.h
+++ b/llvm/include/llvm/CodeGen/TargetLowering.h
@@ -6026,6 +6026,12 @@ class LLVM_ABI TargetLowering : public TargetLoweringBase {
                            SDValue HiLHS = SDValue(),
                            SDValue HiRHS = SDValue()) const;
 
+  /// Expand MUL X, X of an illegal scalar integer type. Only the low bits of
+  /// the result type are computed.
+  /// \param N Node to expand
+  /// \returns The expansion if successful, SDValue() otherwise
+  SDValue expandWideSquare(SDNode *N, SelectionDAG &DAG) const;
+
   /// Calculate full product of LHS and RHS either via a libcall or through
   /// brute force expansion of the multiplication. The expansion works by
   /// splitting the 2 inputs into 4 pieces that we can multiply and add together
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp
index a90bb06fb424dc..2c9f0c90e553bd 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp
@@ -4556,6 +4556,13 @@ void DAGTypeLegalizer::ExpandIntRes_MUL(SDNode *N,
   RTLIB::Libcall LC = RTLIB::getMUL(VT);
   RTLIB::LibcallImpl LCImpl = DAG.getLibcalls().getLibcallImpl(LC);
   if (LCImpl == RTLIB::Unsupported) {
+    // A square needs only half of the limb products.
+    if (N->getOperand(0) == N->getOperand(1)) {
+      if (SDValue Sq = TLI.expandWideSquare(N, DAG)) {
+        SplitInteger(Sq, Lo, Hi);
+        return;
+      }
+    }
     // Perform a wide multiplication where the wide type is the original VT and
     // the 4 parts are the split arguments.
     TLI.forceExpandMultiply(DAG, dl, /*Signed=*/false, Lo, Hi, LL, RL, LH, RH);
diff --git a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
index 7c97dfb2c806a1..e3bbfafa747391 100644
--- a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
@@ -12634,6 +12634,141 @@ void TargetLowering::forceExpandMultiply(SelectionDAG &DAG, const SDLoc &dl,
   }
 }
 
+SDValue TargetLowering::expandWideSquare(SDNode *N, SelectionDAG &DAG) const {
+  assert(N->getOpcode() == ISD::MUL && N->getOperand(0) == N->getOperand(1) &&
+         "Expected a squaring MUL");
+  EVT VT = N->getValueType(0);
+  if (!VT.isScalarInteger())
+    return SDValue();
+  LLVMContext &Ctx = *DAG.getContext();
+  SDLoc dl(N);
+
+  // The limbs have the legal type that VT is ultimately expanded to.
+  EVT LimbVT = getTypeToExpandTo(Ctx, VT);
+  unsigned U = LimbVT.getSizeInBits();
+  unsigned Bits = VT.getSizeInBits();
+  if (Bits < 4 * U)
+    return SDValue();
+  unsigned R = Bits / U; // number of result limbs
+
+  bool HasUMUL_LOHI = isOperationLegalOrCustom(ISD::UMUL_LOHI, LimbVT);
+  bool HasMULHU = isOperationLegalOrCustom(ISD::MULHU, LimbVT);
+  if (!HasUMUL_LOHI && !HasMULHU)
+    return SDValue();
+
+  // X is usually zero-extended (the way to request the full product), so
+  // square it at its real width: only the limbs that may be nonzero take part.
+  SDValue X = N->getOperand(0);
+  unsigned NumLimbs =
+      divideCeil(Bits - DAG.computeKnownBits(X).countMinLeadingZeros(), U);
+  if (NumLimbs < 2)
+    return SDValue();
+  NumLimbs = std::min(NumLimbs, R);
+
+  // Split X into limbs in least-significant-first order.
+  SmallVector<SDValue, 16> Limb(NumLimbs);
+  for (unsigned I = 0; I != NumLimbs; ++I) {
+    SDValue V = DAG.getNode(ISD::SRL, dl, VT, X,
+                            DAG.getShiftAmountConstant(I * U, VT, dl));
+    Limb[I] = DAG.getNode(ISD::TRUNCATE, dl, LimbVT, V);
+  }
+
+  EVT Prod2VT = EVT::getIntegerVT(Ctx, 2 * U);
+  auto LimbsVT = [&](unsigned K) { return EVT::getIntegerVT(Ctx, K * U); };
+
+  // Return A * B as BUILD_PAIR(low, high), or just low.
+  auto MakeProd = [&](SDValue A, SDValue B, bool LowOnly) -> SDValue {
+    if (LowOnly)
+      return DAG.getNode(ISD::MUL, dl, LimbVT, A, B);
+    SDValue PLo, PHi;
+    if (HasUMUL_LOHI) {
+      PLo =
+          DAG.getNode(ISD::UMUL_LOHI, dl, DAG.getVTList(LimbVT, LimbVT), A, B);
+      PHi = PLo.getValue(1);
+    } else {
+      PLo = DAG.getNode(ISD::MUL, dl, LimbVT, A, B);
+      PHi = DAG.getNode(ISD::MULHU, dl, LimbVT, A, B);
+    }
+    return DAG.getNode(ISD::BUILD_PAIR, dl, Prod2VT, PLo, PHi);
+  };
+
+  // Zero-extend V and place it at limb offset Off.
+  auto Place = [&](SDValue V, unsigned Off, EVT WideVT) -> SDValue {
+    V = DAG.getNode(ISD::ZERO_EXTEND, dl, WideVT, V);
+    return DAG.getNode(ISD::SHL, dl, WideVT, V,
+                       DAG.getShiftAmountConstant(Off * U, WideVT, dl));
+  };
+
+  // Concatenate parts that do not overlap.
+  auto Pack = [&](ArrayRef<std::pair<SDValue, unsigned>> Parts, EVT WideVT) {
+    SDNodeFlags Flags;
+    Flags.setDisjoint(true);
+    SDValue Acc;
+    for (const auto &[V, Off] : Parts) {
+      SDValue P = Place(V, Off, WideVT);
+      Acc = Acc ? DAG.getNode(ISD::OR, dl, WideVT, Acc, P, Flags) : P;
+    }
+    return Acc;
+  };
+
+  // Write the 2U-bit product X[i] * X[j] as i.j and the concatenation of
+  // non-overlapping values (least significant on the right) as ||.
+  // For NumLimbs = 4:
+  //
+  //   X * X = S + ((2 * Acc) << U)
+  //   S := 3.3 || 2.2 || 1.1 || 0.0
+  //   Acc := 3.0                                  // D = NumLimbs - 1 = 3
+  //   Acc := (Acc << U) + (3.1 || 2.0)            // D = 2
+  //   Acc := (Acc << U) + (3.2 || 2.1 || 1.0)     // D = 1
+  //
+  // Each parenthesized row is the diagonal j - i = D below: its products
+  // sit at limbs 2i, so they tile without overlap. Acc is accumulated row by
+  // row as in the loop, shifting the sum so far by one limb before adding the
+  // next row. The sum never outgrows the row, because the shifted sum is
+  // one limb narrower than the row and the high limb of a product is at most
+  // 2^U - 2. Row widths are clamped to the R result limbs, which computes the
+  // square modulo 2^Bits.
+  SDValue Acc;
+  for (unsigned D = NumLimbs - 1; D > 0; --D) {
+    unsigned RowLimbs = std::min(2 * (NumLimbs - D), R - D);
+    SmallVector<std::pair<SDValue, unsigned>, 8> Parts;
+    for (unsigned I = 0; I + D < NumLimbs; ++I) {
+      unsigned Pos = 2 * I;
+      if (Pos >= RowLimbs)
+        break;
+      Parts.push_back(
+          {MakeProd(Limb[I], Limb[I + D], /*LowOnly=*/Pos + 1 >= RowLimbs),
+           Pos});
+    }
+    EVT RowVT = LimbsVT(RowLimbs);
+    SDValue Row = Pack(Parts, RowVT);
+    if (!Acc)
+      Acc = Row;
+    else
+      Acc = DAG.getNode(ISD::ADD, dl, RowVT, Place(Acc, 1, RowVT), Row);
+  }
+
+  // Compute (Acc * 2) << U using a shift.
+  unsigned W = std::min(2 * NumLimbs, R); // result limbs that can be nonzero
+  EVT DblVT = LimbsVT(W - 1);
+  SDValue Dbl = DAG.getNode(ISD::SHL, dl, DblVT, Place(Acc, 0, DblVT),
+                            DAG.getShiftAmountConstant(1, DblVT, dl));
+  EVT ZVT = LimbsVT(W);
+  SDValue Cross = Place(Dbl, 1, ZVT);
+
+  // Compute S in the formula above. The squares are created last so that
+  // they are scheduled close to their use.
+  SmallVector<std::pair<SDValue, unsigned>, 8> DiagParts;
+  for (unsigned I = 0; 2 * I < W; ++I)
+    DiagParts.push_back(
+        {MakeProd(Limb[I], Limb[I], /*LowOnly=*/2 * I + 1 >= W), 2 * I});
+  SDValue Z = DAG.getNode(ISD::ADD, dl, ZVT, Cross, Pack(DiagParts, ZVT));
+
+  if (ZVT != VT)
+    Z = DAG.getNode(ISD::ZERO_EXTEND, dl, VT, Z);
+  return Z;
+}
+
 void TargetLowering::forceExpandWideMUL(SelectionDAG &DAG, const SDLoc &dl,
                                         bool Signed, const SDValue LHS,
                                         const SDValue RHS, SDValue &Lo,
diff --git a/llvm/test/CodeGen/AArch64/wide-int-square.ll b/llvm/test/CodeGen/AArch64/wide-int-square.ll
new file mode 100644
index 00000000000000..adfd9efc2d73a1
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/wide-int-square.ll
@@ -0,0 +1,288 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
+; RUN: llc < %s -mtriple=aarch64 | FileCheck %s
+
+; truncated square, 4 x 4 limbs
+define void @sqr_i256(ptr %out, ptr %in) {
+; CHECK-LABEL: sqr_i256:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldp x10, x8, [x1, #8]
+; CHECK-NEXT:    ldr x9, [x1]
+; CHECK-NEXT:    ldr x12, [x1, #24]
+; CHECK-NEXT:    umulh x16, x9, x9
+; CHECK-NEXT:    umulh x11, x9, x8
+; CHECK-NEXT:    mul x13, x10, x8
+; CHECK-NEXT:    madd x11, x9, x12, x11
+; CHECK-NEXT:    umulh x12, x9, x10
+; CHECK-NEXT:    mul x8, x9, x8
+; CHECK-NEXT:    mul x14, x9, x10
+; CHECK-NEXT:    mul x15, x10, x10
+; CHECK-NEXT:    adds x8, x8, x12
+; CHECK-NEXT:    mul x9, x9, x9
+; CHECK-NEXT:    adc x11, x11, x13
+; CHECK-NEXT:    adds x12, x16, x14, lsl #1
+; CHECK-NEXT:    extr x13, x8, x14, #63
+; CHECK-NEXT:    extr x8, x11, x8, #63
+; CHECK-NEXT:    umulh x10, x10, x10
+; CHECK-NEXT:    stp x9, x12, [x0]
+; CHECK-NEXT:    adcs x9, x13, x15
+; CHECK-NEXT:    adc x8, x8, x10
+; CHECK-NEXT:    stp x9, x8, [x0, #16]
+; CHECK-NEXT:    ret
+  %x = load i256, ptr %in
+  %r = mul i256 %x, %x
+  store i256 %r, ptr %out
+  ret void
+}
+
+; full square, 4 limbs -> 8 limbs
+define void @sqr_i512_zext_i256(ptr %out, ptr %in) {
+; CHECK-LABEL: sqr_i512_zext_i256:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    str x19, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset w19, -16
+; CHECK-NEXT:    ldp x13, x8, [x1, #16]
+; CHECK-NEXT:    ldp x11, x9, [x1]
+; CHECK-NEXT:    mul x18, x13, x8
+; CHECK-NEXT:    umulh x15, x11, x13
+; CHECK-NEXT:    mul x16, x11, x8
+; CHECK-NEXT:    mul x12, x9, x8
+; CHECK-NEXT:    umulh x14, x11, x8
+; CHECK-NEXT:    adds x15, x16, x15
+; CHECK-NEXT:    umulh x10, x9, x8
+; CHECK-NEXT:    umulh x3, x11, x9
+; CHECK-NEXT:    adcs x12, x14, x12
+; CHECK-NEXT:    mul x4, x11, x13
+; CHECK-NEXT:    cinc x10, x10, hs
+; CHECK-NEXT:    mul x2, x9, x13
+; CHECK-NEXT:    umulh x1, x9, x13
+; CHECK-NEXT:    adds x16, x4, x3
+; CHECK-NEXT:    umulh x17, x13, x8
+; CHECK-NEXT:    adcs x15, x15, x2
+; CHECK-NEXT:    mul x5, x11, x9
+; CHECK-NEXT:    adcs x12, x12, x1
+; CHECK-NEXT:    umulh x14, x11, x11
+; CHECK-NEXT:    adcs x10, x10, x18
+; CHECK-NEXT:    extr x1, x12, x15, #63
+; CHECK-NEXT:    cinc x17, x17, hs
+; CHECK-NEXT:    extr x15, x15, x16, #63
+; CHECK-NEXT:    extr x12, x10, x12, #63
+; CHECK-NEXT:    mul x19, x9, x9
+; CHECK-NEXT:    extr x10, x17, x10, #63
+; CHECK-NEXT:    extr x18, x16, x5, #63
+; CHECK-NEXT:    mul x11, x11, x11
+; CHECK-NEXT:    adds x14, x14, x5, lsl #1
+; CHECK-NEXT:    umulh x9, x9, x9
+; CHECK-NEXT:    mul x7, x13, x13
+; CHECK-NEXT:    stp x11, x14, [x0]
+; CHECK-NEXT:    adcs x11, x18, x19
+; CHECK-NEXT:    umulh x13, x13, x13
+; CHECK-NEXT:    adcs x9, x15, x9
+; CHECK-NEXT:    mul x6, x8, x8
+; CHECK-NEXT:    stp x11, x9, [x0, #16]
+; CHECK-NEXT:    lsr x9, x17, #63
+; CHECK-NEXT:    adcs x11, x1, x7
+; CHECK-NEXT:    umulh x8, x8, x8
+; CHECK-NEXT:    adcs x12, x12, x13
+; CHECK-NEXT:    stp x11, x12, [x0, #32]
+; CHECK-NEXT:    adcs x10, x10, x6
+; CHECK-NEXT:    adc x8, x9, x8
+; CHECK-NEXT:    stp x10, x8, [x0, #48]
+; CHECK-NEXT:    ldr x19, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEXT:    ret
+  %x = load i256, ptr %in
+  %z = zext i256 %x to i512
+  %r = mul i512 %z, %z
+  store i512 %r, ptr %out
+  ret void
+}
+
+; full square through a promoted type (i768 -> i1024), 6 limbs -> 12 limbs
+define void @sqr_i768_zext_i384(ptr %out, ptr %in) {
+; CHECK-LABEL: sqr_i768_zext_i384:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #128
+; CHECK-NEXT:    stp x29, x30, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT:    stp x28, x27, [sp, #48] // 16-byte Folded Spill
+; CHECK-NEXT:    stp x26, x25, [sp, #64] // 16-byte Folded Spill
+; CHECK-NEXT:    stp x24, x23, [sp, #80] // 16-byte Folded Spill
+; CHECK-NEXT:    stp x22, x21, [sp, #96] // 16-byte Folded Spill
+; CHECK-NEXT:    stp x20, x19, [sp, #112] // 16-byte Folded Spill
+; CHECK-NEXT:    .cfi_def_cfa_offset 128
+; CHECK-NEXT:    .cfi_offset w19, -8
+; CHECK-NEXT:    .cfi_offset w20, -16
+; CHECK-NEXT:    .cfi_offset w21, -24
+; CHECK-NEXT:    .cfi_offset w22, -32
+; CHECK-NEXT:    .cfi_offset w23, -40
+; CHECK-NEXT:    .cfi_offset w24, -48
+; CHECK-NEXT:    .cfi_offset w25, -56
+; CHECK-NEXT:    .cfi_offset w26, -64
+; CHECK-NEXT:    .cfi_offset w27, -72
+; CHECK-NEXT:    .cfi_offset w28, -80
+; CHECK-NEXT:    .cfi_offset w30, -88
+; CHECK-NEXT:    .cfi_offset w29, -96
+; CHECK-NEXT:    ldp x9, x8, [x1, #32]
+; CHECK-NEXT:    ldp x13, x10, [x1]
+; CHECK-NEXT:    ldp x11, x12, [x1, #16]
+; CHECK-NEXT:    umulh x15, x9, x8
+; CHECK-NEXT:    umulh x7, x13, x9
+; CHECK-NEXT:    mul x19, x13, x8
+; CHECK-NEXT:    mul x5, x10, x8
+; CHECK-NEXT:    umulh x6, x13, x8
+; CHECK-NEXT:    adds x7, x19, x7
+; CHECK-NEXT:    umulh x17, x10, x8
+; CHECK-NEXT:    umulh x23, x13, x12
+; CHECK-NEXT:    adcs x5, x6, x5
+; CHECK-NEXT:    mul x24, x13, x9
+; CHECK-NEXT:    cinc x17, x17, hs
+; CHECK-NEXT:    mul x14, x9, x8
+; CHECK-NEXT:    mul x22, x10, x9
+; CHECK-NEXT:    adds x19, x24, x23
+; CHECK-NEXT:    umulh x21, x10, x9
+; CHECK-NEXT:    stp x14, x15, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    mul x20, x11, x8
+; CHECK-NEXT:    adcs x7, x7, x22
+; CHECK-NEXT:    umulh x3, x11, x8
+; CHECK-NEXT:    adcs x5, x5, x21
+; CHECK-NEXT:    umulh x29, x13, x11
+; CHECK-NEXT:    adcs x20, x17, x20
+; CHECK-NEXT:    mul x30, x13, x12
+; CHECK-NEXT:    cinc x21, x3, hs
+; CHECK-NEXT:    mul x28, x10, x12
+; CHECK-NEXT:    umulh x15, x12, x9
+; CHECK-NEXT:    adds x23, x30, x29
+; CHECK-NEXT:    ldp x29, x30, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT:    mul x14, x12, x9
+; CHECK-NEXT:    adcs x19, x19, x28
+; CHECK-NEXT:    umulh x27, x10, x12
+; CHECK-NEXT:    mul x26, x11, x9
+; CHECK-NEXT:    stp x14, x15, [sp] // 16-byte Folded Spill
+; CHECK-NEXT:    umulh x25, x11, x9
+; CHECK-NEXT:    adcs x7, x7, x27
+; CHECK-NEXT:    ldp x28, x27, [sp, #48] // 16-byte Folded Reload
+; CHECK-NEXT:    mul x4, x12, x8
+; CHECK-NEXT:    adcs x5, x5, x26
+; CHECK-NEXT:    umulh x2, x12, x8
+; CHECK-NEXT:    adcs x20, x20, x25
+; CHECK-NEXT:    ldp x26, x25, [sp, #64] // 16-byte Folded Reload
+; CHECK-NEXT:    umulh x14, x13, x10
+; CHECK-NEXT:    adcs x4, x21, x4
+; CHECK-NEXT:    mul x6, x13, x11
+; CHECK-NEXT:    cinc x2, x2, hs
+; CHECK-NEXT:    mul x15, x10, x11
+; CHECK-NEXT:    umulh x16, x10, x11
+; CHECK-NEXT:    adds x14, x6, x14
+; CHECK-NEXT:    mul x18, x11, x12
+; CHECK-NEXT:    adcs x15, x23, x15
+; CHECK-NEXT:    umulh x1, x11, x12
+; CHECK-NEXT:    adcs x16, x19, x16
+; CHECK-NEXT:    mul x22, x13, x10
+; CHECK-NEXT:    adcs x18, x7, x18
+; CHECK-NEXT:    umulh x6, x13, x13
+; CHECK-NEXT:    adcs x1, x5, x1
+; CHECK-NEXT:    ldp x5, x19, [sp] // 16-byte Folded Reload
+; CHECK-NEXT:    mul x7, x10, x10
+; CHECK-NEXT:    adcs x5, x20, x5
+; CHECK-NEXT:    umulh x10, x10, x10
+; CHECK-NEXT:    adcs x4, x4, x19
+; CHECK-NEXT:    ldr x19, [sp, #16] // 8-byte Reload
+; CHECK-NEXT:    mul x21, x11, x11
+; CHECK-NEXT:    extr x20, x4, x5, #63
+; CHECK-NEXT:    adcs x2, x2, x19
+; CHECK-NEXT:    ldr x19, [sp, #24] // 8-byte Reload
+; CHECK-NEXT:    mul x13, x13, x13
+; CHECK-NEXT:    cinc x19, x19, hs
+; CHECK-NEXT:    adds x6, x6, x22, lsl #1
+; CHECK-NEXT:    extr x22, x14, x22, #63
+; CHECK-NEXT:    umulh x11, x11, x11
+; CHECK-NEXT:    extr x14, x15, x14, #63
+; CHECK-NEXT:    extr x15, x16, x15, #63
+; CHECK-NEXT:    adcs x7, x22, x7
+; CHECK-NEXT:    extr x16, x18, x16, #63
+; CHECK-NEXT:    mul x24, x12, x12
+; CHECK-NEXT:    adcs x10, x14, x10
+; CHECK-NEXT:    stp x13, x6, [x0]
+; CHECK-NEXT:    extr x13, x1, x18, #63
+; CHECK-NEXT:    adcs x14, x15, x21
+; CHECK-NEXT:    umulh x12, x12, x12
+; CHECK-NEXT:    stp x7, x10, [x0, #16]
+; CHECK-NEXT:    extr x10, x5, x1, #63
+; CHECK-NEXT:    adcs x11, x16, x11
+; CHECK-NEXT:    ldp x22, x21, [sp, #96] // 16-byte Folded Reload
+; CHECK-NEXT:    mul x3, x9, x9
+; CHECK-NEXT:    stp x14, x11, [x0, #32]
+; CHECK-NEXT:    extr x11, x2, x4, #63
+; CHECK-NEXT:    adcs x13, x13, x24
+; CHECK-NEXT:    ldp x24, x23, [sp, #80] // 16-byte Folded Reload
+; CHECK-NEXT:    umulh x9, x9, x9
+; CHECK-NEXT:    adcs x10, x10, x12
+; CHECK-NEXT:    extr x12, x19, x2, #63
+; CHECK-NEXT:    mul x17, x8, x8
+; CHECK-NEXT:    stp x13, x10, [x0, #48]
+; CHECK-NEXT:    lsr x10, x19, #63
+; CHECK-NEXT:    adcs x13, x20, x3
+; CHECK-NEXT:    ldp x20, x19, [sp, #112] // 16-byte Folded Reload
+; CHECK-NEXT:    umulh x8, x8, x8
+; CHECK-NEXT:    adcs x9, x11, x9
+; CHECK-NEXT:    stp x13, x9, [x0, #64]
+; CHECK-NEXT:    adcs x11, x12, x17
+; CHECK-NEXT:    adc x8, x10, x8
+; CHECK-NEXT:    stp x11, x8, [x0, #80]
+; CHECK-NEXT:    add sp, sp, #128
+; CHECK-NEXT:    ret
+  %x = load i384, ptr %in
+  %z = zext i384 %x to i768
+  %r = mul i768 %z, %z
+  store i768 %r, ptr %out
+  ret void
+}
+
+; sign-extended operand: no known-zero limbs, truncated 4 x 4
+define void @sqr_i256_sext_i128(ptr %out, ptr %in) {
+; CHECK-LABEL: sqr_i256_sext_i128:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldp x9, x8, [x1]
+; CHECK-NEXT:    asr x10, x8, #63
+; CHECK-NEXT:    umulh x13, x9, x8
+; CHECK-NEXT:    umulh x11, x9, x10
+; CHECK-NEXT:    mul x12, x9, x10
+; CHECK-NEXT:    mul x10, x8, x10
+; CHECK-NEXT:    mul x14, x9, x8
+; CHECK-NEXT:    add x11, x12, x11
+; CHECK-NEXT:    adds x12, x12, x13
+; CHECK-NEXT:    umulh x16, x9, x9
+; CHECK-NEXT:    adc x10, x11, x10
+; CHECK-NEXT:    mul x15, x8, x8
+; CHECK-NEXT:    extr x10, x10, x12, #63
+; CHECK-NEXT:    extr x13, x12, x14, #63
+; CHECK-NEXT:    mul x9, x9, x9
+; CHECK-NEXT:    adds x11, x16, x14, lsl #1
+; CHECK-NEXT:    umulh x8, x8, x8
+; CHECK-NEXT:    stp x9, x11, [x0]
+; CHECK-NEXT:    adcs x9, x13, x15
+; CHECK-NEXT:    adc x8, x10, x8
+; CHECK-NEXT:    stp x9, x8, [x0, #16]
+; CHECK-NEXT:    ret
+  %x = load i128, ptr %in
+  %z = sext i128 %x to i256
+  %r = mul i256 %z, %z
+  store i256 %r, ptr %out
+  ret void
+}
+
+; a single limb: one product, not expanded here
+define void @sqr_i256_zext_i64(ptr %out, ptr %in) {
+; CHECK-LABEL: sqr_i256_zext_i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr x8, [x1]
+; CHECK-NEXT:    stp xzr, xzr, [x0, #16]
+; CHECK-NEXT:    umulh x9, x8, x8
+; CHECK-NEXT:    mul x8, x8, x8
+; CHECK-NEXT:    stp x8, x9, [x0]
+; CHECK-NEXT:    ret
+  %x = load i64, ptr %in
+  %z = zext i64 %x to i256
+  %r = mul i256 %z, %z
+  store i256 %r, ptr %out
+  ret void
+}
diff --git a/llvm/test/CodeGen/RISCV/wide-int-square.ll b/llvm/test/CodeGen/RISCV/wide-int-square.ll
new file mode 100644
index 00000000000000..c66d4249da3625
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/wide-int-square.ll
@@ -0,0 +1,723 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
+; RUN: llc < %s -mtriple=riscv64 -mattr=+m | FileCheck %s
+
+; truncated square, 4 x 4 limbs
+define void @sqr_i256(ptr %out, ptr %in) {
+; CHECK-LABEL: sqr_i256:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    ld a2, 0(a1)
+; CHECK-NEXT:    ld a3, 8(a1)
+; CHECK-NEXT:    ld a4, 16(a1)
+; CHECK-NEXT:    ld a1, 24(a1)
+; CHECK-NEXT:    mulhu a5, a2, a3
+; CHECK-NEXT:    mul a6, a2, a4
+; CHECK-NEXT:    mulhu a7, a2, a4
+; CHECK-NEXT:    mul a1, a2, a1
+; CHECK-NEXT:    mul t0, a2, a3
+; CHECK-NEXT:    mul a4, a3, a4
+; CHECK-NEXT:    mul t1, a3, a3
+; CHECK-NEXT:    mulhu t2, a2, a2
+; CHECK-NEXT:    mulhu a3, a3, a3
+; CHECK-NEXT:    mul a2, a2, a2
+; CHECK-NEXT:    add a5, a6, a5
+; CHECK-NEXT:    add a1, a1, a7
+; CHECK-NEXT:    srli a7, t0, 63
+; CHECK-NEXT:    slli t0, t0, 1
+; CHECK-NEXT:    add a1, a1, a4
+; CHECK-NEXT:    slli a4, a5, 1
+; CHECK-NEXT:    sltu a6, a5, a6
+; CHECK-NEXT:    srli a5, a5, 63
+; CHECK-NEXT:    add t2, t0, t2
+; CHECK-NEXT:    or a4, a4, a7
+; CHECK-NEXT:    add a1, a1, a6
+; CHECK-NEXT:    sltu a6, t2, t0
+; CHECK-NEXT:    add t1, a4, t1
+; CHECK-NEXT:    slli a1, a1, 1
+; CHECK-NEXT:    sltu a4, t1, a4
+; CHECK-NEXT:    or a1, a1, a5
+; CHECK-NEXT:    add a6, t1, a6
+; CHECK-NEXT:    add a1, a1, a3
+; CHECK-NEXT:    sltu a3, a6, t1
+; CHECK-NEXT:    add a1, a1, a4
+; CHECK-NEXT:    add a1, a1, a3
+; CHECK-NEXT:    sd a2, 0(a0)
+; CHECK-NEXT:    sd t2, 8(a0)
+; CHECK-NEXT:    sd a6, 16(a0)
+; CHECK-NEXT:    sd a1, 24(a0)
+; CHECK-NEXT:    ret
+  %x = load i256, ptr %in
+  %r = mul i256 %x, %x
+  store i256 %r, ptr %out
+  ret void
+}
+
+; full square, 4 limbs -> 8 limbs
+define void @sqr_i512_zext_i256(ptr %out, ptr %in) {
+; CHECK-LABEL: sqr_i512_zext_i256:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -64
+; CHECK-NEXT:    .cfi_def_cfa_offset 64
+; CHECK-NEXT:    sd s0, 56(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s1, 48(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s2, 40(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s3, 32(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s4, 24(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s5, 16(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s6, 8(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s7, 0(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    .cfi_offset s0, -8
+; CHECK-NEXT:    .cfi_offset s1, -16
+; CHECK-NEXT:    .cfi_offset s2, -24
+; CHECK-NEXT:    .cfi_offset s3, -32
+; CHECK-NEXT:    .cfi_offset s4, -40
+; CHECK-NEXT:    .cfi_offset s5, -48
+; CHECK-NEXT:    .cfi_offset s6, -56
+; CHECK-NEXT:    .cfi_offset s7, -64
+; CHECK-NEXT:    ld a4, 16(a1)
+; CHECK-NEXT:    ld a3, 24(a1)
+; CHECK-NEXT:    ld a2, 0(a1)
+; CHECK-NEXT:    ld a5, 8(a1)
+; CHECK-NEXT:    mulhu a1, a2, a4
+; CHECK-NEXT:    mul a6, a2, a3
+; CHECK-NEXT:    mul t3, a5, a3
+; CHECK-NEXT:    mulhu t1, a2, a3
+; CHECK-NEXT:    mulhu a7, a2, a5
+; CHECK-NEXT:    mul t0, a2, a4
+; CHECK-NEXT:    mul t2, a5, a4
+; CHECK-NEXT:    mulhu t4, a5, a4
+; CHECK-NEXT:    add a1, a6, a1
+; CHECK-NEXT:    add t3, t1, t3
+; CHECK-NEXT:    add a7, t0, a7
+; CHECK-NEXT:    add t6, a1, t2
+; CHECK-NEXT:    sltu t5, a1, a6
+; CHECK-NEXT:    sltu a6, t6, a1
+; CHECK-NEXT:    add t5, t3, t5
+; CHECK-NEXT:    sltu t0, a7, t0
+; CHECK-NEXT:    mul s0, a4, a4
+; CHECK-NEXT:    add t4, t5, t4
+; CHECK-NEXT:    add t2, t6, t0
+; CHECK-NEXT:    add a6, t4, a6
+; CHECK-NEXT:    sltu t4, t2, t6
+; CHECK-NEXT:    add t4, a6, t4
+; CHECK-NEXT:    slli a6, t4, 1
+; CHECK-NEXT:    srli t6, t2, 63
+; CHECK-NEXT:    or t6, a6, t6
+; CHECK-NEXT:    add a6, t6, s0
+; CHECK-NEXT:    beq t4, t5, .LBB1_2
+; CHECK-NEXT:  # %bb.1:
+; CHECK-NEXT:    sltu s0, t4, t5
+; CHECK-NEXT:    j .LBB1_3
+; CHECK-NEXT:  .LBB1_2:
+; CHECK-NEXT:    sltu s0, t2, a1
+; CHECK-NEXT:  .LBB1_3:
+; CHECK-NEXT:    xor s1, t4, t5
+; CHECK-NEXT:    xor a1, t2, a1
+; CHECK-NEXT:    or s1, a1, s1
+; CHECK-NEXT:    sltu a1, a6, t6
+; CHECK-NEXT:    beqz s1, .LBB1_5
+; CHECK-NEXT:  # %bb.4:
+; CHECK-NEXT:    mv t0, s0
+; CHECK-NEXT:  .LBB1_5:
+; CHECK-NEXT:    mulhu t6, a5, a3
+; CHECK-NEXT:    sltu t5, t5, t3
+; CHECK-NEXT:    mul s0, a4, a3
+; CHECK-NEXT:    sltu t1, t3, t1
+; CHECK-NEXT:    mul s1, a2, a5
+; CHECK-NEXT:    srli s2, t4, 63
+; CHECK-NEXT:    mulhu s3, a4, a4
+; CHECK-NEXT:    slli t4, a7, 1
+; CHECK-NEXT:    mul s4, a5, a5
+; CHECK-NEXT:    slli s5, t2, 1
+; CHECK-NEXT:    add t1, t1, t5
+; CHECK-NEXT:    mulhu s6, a2, a2
+; CHECK-NEXT:    add t1, t6, t1
+; CHECK-NEXT:    mulhu s7, a5, a5
+; CHECK-NEXT:    add t2, t1, s0
+; CHECK-NEXT:    srli a5, s1, 63
+; CHECK-NEXT:    add t3, t2, t0
+; CHECK-NEXT:    or t4, t4, a5
+; CHECK-NEXT:    slli a5, t3, 1
+; CHECK-NEXT:    add s4, t4, s4
+; CHECK-NEXT:    or t5, a5, s2
+; CHECK-NEXT:    sltu t0, s4, t4
+; CHECK-NEXT:    slli t6, s1, 1
+; CHECK-NEXT:    srli a7, a7, 63
+; CHECK-NEXT:    add a5, t6, s6
+; CHECK-NEXT:    or s1, s5, a7
+; CHECK-NEXT:    sltu t6, a5, t6
+; CHECK-NEXT:    add s7, s1, s7
+; CHECK-NEXT:    add a7, s4, t6
+; CHECK-NEXT:    add t0, s7, t0
+; CHECK-NEXT:    sltu s0, a7, s4
+; CHECK-NEXT:    add s3, s3, a1
+; CHECK-NEXT:    add t0, t0, s0
+; CHECK-NEXT:    add s0, t5, s3
+; CHECK-NEXT:    beq t0, s1, .LBB1_7
+; CHECK-NEXT:  # %bb.6:
+; CHECK-NEXT:    sltu s2, t0, s1
+; CHECK-NEXT:    j .LBB1_8
+; CHECK-NEXT:  .LBB1_7:
+; CHECK-NEXT:    sltu s2, a7, t4
+; CHECK-NEXT:  .LBB1_8:
+; CHECK-NEXT:    xor s1, t0, s1
+; CHECK-NEXT:    xor t4, a7, t4
+; CHECK-NEXT:    or t4, t4, s1
+; CHECK-NEXT:    beqz t4, .LBB1_10
+; CHECK-NEXT:  # %bb.9:
+; CHECK-NEXT:    mv t6, s2
+; CHECK-NEXT:  .LBB1_10:
+; CHECK-NEXT:    add t4, a6, t6
+; CHECK-NEXT:    sltu t6, t4, a6
+; CHECK-NEXT:    add a6, s0, t6
+; CHECK-NEXT:    sltu s1, a6, s0
+; CHECK-NEXT:    and t6, t6, s1
+; CHECK-NEXT:    beq s0, t5, .LBB1_12
+; CHECK-NEXT:  # %bb.11:
+; CHECK-NEXT:    sltu a1, s0, t5
+; CHECK-NEXT:  .LBB1_12:
+; CHECK-NEXT:    mulhu a4, a4, a3
+; CHECK-NEXT:    mul t5, a3, a3
+; CHECK-NEXT:    mulhu a3, a3, a3
+; CHECK-NEXT:    sltu t1, t2, t1
+; CHECK-NEXT:    sltu t2, t3, t2
+; CHECK-NEXT:    add t1, t1, t2
+; CHECK-NEXT:    srli t2, t3, 63
+; CHECK-NEXT:    add a4, a4, t1
+; CHECK-NEXT:    mul a2, a2, a2
+; CHECK-NEXT:    slli t1, a4, 1
+; CHECK-NEXT:    srli a4, a4, 63
+; CHECK-NEXT:    or t1, t1, t2
+; CHECK-NEXT:    add a3, a4, a3
+; CHECK-NEXT:    add t5, t1, t5
+; CHECK-NEXT:    add a1, t5, a1
+; CHECK-NEXT:    sltu a4, t5, t1
+; CHECK-NEXT:    add t6, a1, t6
+; CHECK-NEXT:    add a3, a3, a4
+; CHECK-NEXT:    sltu a4, a1, t5
+; CHECK-NEXT:    sltu a1, t6, a1
+; CHECK-NEXT:    add a3, a3, a4
+; CHECK-NEXT:    sd a2, 0(a0)
+; CHECK-NEXT:    sd a5, 8(a0)
+; CHECK-NEXT:    sd a7, 16(a0)
+; CHECK-NEXT:    sd t0, 24(a0)
+; CHECK-NEXT:    add a1, a3, a1
+; CHECK-NEXT:    sd t4, 32(a0)
+; CHECK-NEXT:    sd a6, 40(a0)
+; CHECK-NEXT:    sd t6, 48(a0)
+; CHECK-NEXT:    sd a1, 56(a0)
+; CHECK-NEXT:    ld s0, 56(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s1, 48(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s2, 40(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s3, 32(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s4, 24(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s5, 16(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s6, 8(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s7, 0(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    .cfi_restore s0
+; CHECK-NEXT:    .cfi_restore s1
+; CHECK-NEXT:    .cfi_restore s2
+; CHECK-NEXT:    .cfi_restore s3
+; CHECK-NEXT:    .cfi_restore s4
+; CHECK-NEXT:    .cfi_restore s5
+; CHECK-NEXT:    .cfi_restore s6
+; CHECK-NEXT:    .cfi_restore s7
+; CHECK-NEXT:    addi sp, sp, 64
+; CHECK-NEXT:    .cfi_def_cfa_offset 0
+; CHECK-NEXT:    ret
+  %x = load i256, ptr %in
+  %z = zext i256 %x to i512
+  %r = mul i512 %z, %z
+  store i512 %r, ptr %out
+  ret void
+}
+
+; full square through a promoted type (i768 -> i1024), 6 limbs -> 12 limbs
+define void @sqr_i768_zext_i384(ptr %out, ptr %in) {
+; CHECK-LABEL: sqr_i768_zext_i384:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addi sp, sp, -112
+; CHECK-NEXT:    .cfi_def_cfa_offset 112
+; CHECK-NEXT:    sd ra, 104(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s0, 96(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s1, 88(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s2, 80(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s3, 72(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s4, 64(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s5, 56(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s6, 48(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s7, 40(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s8, 32(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s9, 24(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s10, 16(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    sd s11, 8(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    .cfi_offset ra, -8
+; CHECK-NEXT:    .cfi_offset s0, -16
+; CHECK-NEXT:    .cfi_offset s1, -24
+; CHECK-NEXT:    .cfi_offset s2, -32
+; CHECK-NEXT:    .cfi_offset s3, -40
+; CHECK-NEXT:    .cfi_offset s4, -48
+; CHECK-NEXT:    .cfi_offset s5, -56
+; CHECK-NEXT:    .cfi_offset s6, -64
+; CHECK-NEXT:    .cfi_offset s7, -72
+; CHECK-NEXT:    .cfi_offset s8, -80
+; CHECK-NEXT:    .cfi_offset s9, -88
+; CHECK-NEXT:    .cfi_offset s10, -96
+; CHECK-NEXT:    .cfi_offset s11, -104
+; CHECK-NEXT:    ld a5, 24(a1)
+; CHECK-NEXT:    ld a2, 0(a1)
+; CHECK-NEXT:    ld a4, 32(a1)
+; CHECK-NEXT:    ld a3, 40(a1)
+; CHECK-NEXT:    ld a6, 16(a1)
+; CHECK-NEXT:    ld a1, 8(a1)
+; CHECK-NEXT:    mulhu t4, a2, a5
+; CHECK-NEXT:    mul t0, a2, a4
+; CHECK-NEXT:    mulhu s2, a2, a4
+; CHECK-NEXT:    mul t1, a2, a3
+; CHECK-NEXT:    mulhu a7, a2, a6
+; CHECK-NEXT:    mul t3, a2, a5
+; CHECK-NEXT:    mul t2, a1, a5
+; CHECK-NEXT:    mul t6, a1, a4
+; CHECK-NEXT:    mulhu s0, a1, a5
+; CHECK-NEXT:    mul s4, a6, a5
+; CHECK-NEXT:    add t4, t0, t4
+; CHECK-NEXT:    add s2, t1, s2
+; CHECK-NEXT:    add t5, t3, a7
+; CHECK-NEXT:    add s5, t4, t2
+; CHECK-NEXT:    sltu a7, t4, t0
+; CHECK-NEXT:    add t2, s2, t6
+; CHECK-NEXT:    sltu t0, s5, t4
+; CHECK-NEXT:    add s3, t2, a7
+; CHECK-NEXT:    sltu s1, t5, t3
+; CHECK-NEXT:    add s0, s3, s0
+; CHECK-NEXT:    add s6, s5, s1
+; CHECK-NEXT:    add t0, s0, t0
+; CHECK-NEXT:    sltu t3, s6, s5
+; CHECK-NEXT:    add t3, t0, t3
+; CHECK-NEXT:    add t0, t3, s4
+; CHECK-NEXT:    beq t3, s3, .LBB2_2
+; CHECK-NEXT:  # %bb.1:
+; CHECK-NEXT:    sltu t6, t3, s3
+; CHECK-NEXT:    j .LBB2_3
+; CHECK-NEXT:  .LBB2_2:
+; CHECK-NEXT:    sltu t6, s6, t4
+; CHECK-NEXT:  .LBB2_3:
+; CHECK-NEXT:    sd a7, 0(sp) # 8-byte Folded Spill
+; CHECK-NEXT:    xor s0, t3, s3
+; CHECK-NEXT:    xor t4, s6, t4
+; CHECK-NEXT:    or t4, t4, s0
+; CHECK-NEXT:    sltu s0, t0, t3
+; CHECK-NEXT:    beqz t4, .LBB2_5
+; CHECK-NEXT:  # %bb.4:
+; CHECK-NEXT:    mv s1, t6
+; CHECK-NEXT:  .LBB2_5:
+; CHECK-NEXT:    mul s7, a1, a3
+; CHECK-NEXT:    mulhu s4, a2, a3
+; CHECK-NEXT:    mulhu t4, a1, a4
+; CHECK-NEXT:    sltu t6, t2, s2
+; CHECK-NEXT:    mul s5, a1, a6
+; CHECK-NEXT:    sltu s8, s2, t1
+; CHECK-NEXT:    mul t1, a6, a4
+; CHECK-NEXT:    mulhu s10, a2, a1
+; CHECK-NEXT:    mul s11, a2, a6
+; CHECK-NEXT:    sltu t2, s3, t2
+; CHECK-NEXT:    mulhu ra, a1, a6
+; CHECK-NEXT:    add s7, s4, s7
+; CHECK-NEXT:    add t4, t4, t6
+; CHECK-NEXT:    mulhu t6, a6, a5
+; CHECK-NEXT:    add s8, s7, s8
+; CHECK-NEXT:    add t2, t4, t2
+; CHECK-NEXT:    add a7, t5, s5
+; CHECK-NEXT:    add s9, s8, t2
+; CHECK-NEXT:    sltu t4, a7, t5
+; CHECK-NEXT:    add s5, s9, t1
+; CHECK-NEXT:    add t2, s11, s10
+; CHECK-NEXT:    add ra, s6, ra
+; CHECK-NEXT:    sltu t1, t2, s11
+; CHECK-NEXT:    add ra, ra, t4
+; CHECK-NEXT:    add t4, a7, t1
+; CHECK-NEXT:    add s1, s5, s1
+; CHECK-NEXT:    sltu a7, t4, a7
+; CHECK-NEXT:    add s10, t6, s0
+; CHECK-NEXT:    add t6, ra, a7
+; CHECK-NEXT:    add s10, s1, s10
+; CHECK-NEXT:    beq t6, s6, .LBB2_7
+; CHECK-NEXT:  # %bb.6:
+; CHECK-NEXT:    sltu s11, t6, s6
+; CHECK-NEXT:    j .LBB2_8
+; CHECK-NEXT:  .LBB2_7:
+; CHECK-NEXT:    sltu s11, t4, t5
+; CHECK-NEXT:  .LBB2_8:
+; CHECK-NEXT:    xor a7, t6, s6
+; CHECK-NEXT:    xor t5, t4, t5
+; CHECK-NEXT:    or a7, t5, a7
+; CHECK-NEXT:    beqz a7, .LBB2_10
+; CHECK-NEXT:  # %bb.9:
+; CHECK-NEXT:    mv t1, s11
+; CHECK-NEXT:  .LBB2_10:
+; CHECK-NEXT:    add a7, t0, t1
+; CHECK-NEXT:    sltu s6, a7, t0
+; CHECK-NEXT:    add t0, s10, s6
+; CHECK-NEXT:    beq s10, s1, .LBB2_12
+; CHECK-NEXT:  # %bb.11:
+; CHECK-NEXT:    sltu s0, s10, s1
+; CHECK-NEXT:  .LBB2_12:
+; CHECK-NEXT:    sltu s11, t0, s10
+; CHECK-NEXT:    beq s9, s8, .LBB2_14
+; CHECK-NEXT:  # %bb.13:
+; CHECK-NEXT:    sltu s10, s9, s8
+; CHECK-NEXT:    j .LBB2_15
+; CHECK-NEXT:  .LBB2_14:
+; CHECK-NEXT:    sltu s10, s3, s2
+; CHECK-NEXT:  .LBB2_15:
+; CHECK-NEXT:    xor t5, s3, s2
+; CHECK-NEXT:    xor s2, s9, s8
+; CHECK-NEXT:    or t5, t5, s2
+; CHECK-NEXT:    and s6, s6, s11
+; CHECK-NEXT:    sltu s2, s5, s9
+; CHECK-NEXT:    ld s11, 0(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    beqz t5, .LBB2_17
+; CHECK-NEXT:  # %bb.16:
+; CHECK-NEXT:    mv s11, s10
+; CHECK-NEXT:  .LBB2_17:
+; CHECK-NEXT:    mulhu t5, a1, a3
+; CHECK-NEXT:    mul s3, a6, a3
+; CHECK-NEXT:    mulhu s9, a6, a4
+; CHECK-NEXT:    sltu s8, s8, s7
+; CHECK-NEXT:    sltu s4, s7, s4
+; CHECK-NEXT:    add s4, s4, s8
+; CHECK-NEXT:    mul s8, a5, a4
+; CHECK-NEXT:    add s4, t5, s4
+; CHECK-NEXT:    add s7, s4, s3
+; CHECK-NEXT:    add s10, s7, s11
+; CHECK-NEXT:    add s9, s9, s2
+; CHECK-NEXT:    add s11, s10, s9
+; CHECK-NEXT:    sltu s5, s1, s5
+; CHECK-NEXT:    add s3, s11, s5
+; CHECK-NEXT:    add s8, s3, s8
+; CHECK-NEXT:    add s0, s8, s0
+; CHECK-NEXT:    add t5, s0, s6
+; CHECK-NEXT:    sltu s6, s3, s11
+; CHECK-NEXT:    sltu s9, t5, s0
+; CHECK-NEXT:    and s6, s5, s6
+; CHECK-NEXT:    beq s11, s10, .LBB2_19
+; CHECK-NEXT:  # %bb.18:
+; CHECK-NEXT:    sltu s2, s11, s10
+; CHECK-NEXT:  .LBB2_19:
+; CHECK-NEXT:    mulhu s5, a6, a3
+; CHECK-NEXT:    mul s11, a5, a3
+; CHECK-NEXT:    mulhu ra, a5, a4
+; CHECK-NEXT:    sltu s10, s10, s7
+; CHECK-NEXT:    sltu s4, s7, s4
+; CHECK-NEXT:    add s4, s4, s10
+; CHECK-NEXT:    add s4, s5, s4
+; CHECK-NEXT:    sltu s7, s8, s3
+; CHECK-NEXT:    add s5, s4, s11
+; CHECK-NEXT:    add s7, ra, s7
+; CHECK-NEXT:    sltu s0, s0, s8
+; CHECK-NEXT:    add s2, s5, s2
+; CHECK-NEXT:    add s0, s7, s0
+; CHECK-NEXT:    add s6, s2, s6
+; CHECK-NEXT:    add s0, s0, s9
+; CHECK-NEXT:    add s0, s6, s0
+; CHECK-NEXT:    beq s0, s6, .LBB2_21
+; CHECK-NEXT:  # %bb.20:
+; CHECK-NEXT:    sltu s8, s0, s6
+; CHECK-NEXT:    j .LBB2_22
+; CHECK-NEXT:  .LBB2_21:
+; CHECK-NEXT:    sltu s8, t5, s3
+; CHECK-NEXT:  .LBB2_22:
+; CHECK-NEXT:    xor s7, s0, s6
+; CHECK-NEXT:    xor s3, t5, s3
+; CHECK-NEXT:    or s10, s3, s7
+; CHECK-NEXT:    beq t0, s1, .LBB2_24
+; CHECK-NEXT:  # %bb.23:
+; CHECK-NEXT:    sltu s9, t0, s1
+; CHECK-NEXT:    bnez s10, .LBB2_25
+; CHECK-NEXT:    j .LBB2_26
+; CHECK-NEXT:  .LBB2_24:
+; CHECK-NEXT:    sltu s9, a7, t3
+; CHECK-NEXT:    beqz s10, .LBB2_26
+; CHECK-NEXT:  .LBB2_25:
+; CHECK-NEXT:    mv s9, s8
+; CHECK-NEXT:  .LBB2_26:
+; CHECK-NEXT:    xor s1, t0, s1
+; CHECK-NEXT:    xor t3, a7, t3
+; CHECK-NEXT:    or s1, s1, s7
+; CHECK-NEXT:    or t3, t3, s3
+; CHECK-NEXT:    or t3, t3, s1
+; CHECK-NEXT:    beqz t3, .LBB2_28
+; CHECK-NEXT:  # %bb.27:
+; CHECK-NEXT:    mv t1, s9
+; CHECK-NEXT:  .LBB2_28:
+; CHECK-NEXT:    slli s3, s0, 1
+; CHECK-NEXT:    sltu s7, s6, s2
+; CHECK-NEXT:    sltu s1, s2, s5
+; CHECK-NEXT:    sltu s2, s5, s4
+; CHECK-NEXT:    mulhu t3, a5, a3
+; CHECK-NEXT:    add s1, s2, s1
+; CHECK-NEXT:    add s7, s1, s7
+; CHECK-NEXT:    mul s4, a2, a1
+; CHECK-NEXT:    mul s5, a4, a3
+; CHECK-NEXT:    slli s1, t6, 1
+; CHECK-NEXT:    srli s2, t4, 63
+; CHECK-NEXT:    mul s6, a1, a1
+; CHECK-NEXT:    slli s8, a7, 1
+; CHECK-NEXT:    srli t6, t6, 63
+; CHECK-NEXT:    or s1, s1, s2
+; CHECK-NEXT:    or s2, s8, t6
+; CHECK-NEXT:    slli t6, t2, 1
+; CHECK-NEXT:    mulhu s8, a2, a2
+; CHECK-NEXT:    srli s9, s4, 63
+; CHECK-NEXT:    mulhu s10, a1, a1
+; CHECK-NEXT:    mul s11, a6, a6
+; CHECK-NEXT:    or t6, t6, s9
+; CHECK-NEXT:    mulhu ra, a6, a6
+; CHECK-NEXT:    add s6, t6, s6
+; CHECK-NEXT:    sltu a6, s6, t6
+; CHECK-NEXT:    slli t4, t4, 1
+; CHECK-NEXT:    srli a1, t2, 63
+; CHECK-NEXT:    slli s4, s4, 1
+; CHECK-NEXT:    or t4, t4, a1
+; CHECK-NEXT:    add a1, s4, s8
+; CHECK-NEXT:    add s10, t4, s10
+; CHECK-NEXT:    sltu s4, a1, s4
+; CHECK-NEXT:    add t2, s10, a6
+; CHECK-NEXT:    add a6, s6, s4
+; CHECK-NEXT:    srli s10, t5, 63
+; CHECK-NEXT:    sltu s6, a6, s6
+; CHECK-NEXT:    add s9, s1, s11
+; CHECK-NEXT:    add t2, t2, s6
+; CHECK-NEXT:    sltu s6, s9, s1
+; CHECK-NEXT:    add s8, s2, ra
+; CHECK-NEXT:    add t3, t3, s7
+; CHECK-NEXT:    add s8, s8, s6
+; CHECK-NEXT:    beq t2, t4, .LBB2_30
+; CHECK-NEXT:  # %bb.29:
+; CHECK-NEXT:    sltu ra, t2, t4
+; CHECK-NEXT:    j .LBB2_31
+; CHECK-NEXT:  .LBB2_30:
+; CHECK-NEXT:    sltu ra, a6, t6
+; CHECK-NEXT:  .LBB2_31:
+; CHECK-NEXT:    mul s11, a4, a4
+; CHECK-NEXT:    xor t4, t2, t4
+; CHECK-NEXT:    xor t6, a6, t6
+; CHECK-NEXT:    or t4, t6, t4
+; CHECK-NEXT:    or s7, s3, s10
+; CHECK-NEXT:    add s3, t3, s5
+; CHECK-NEXT:    beqz t4, .LBB2_33
+; CHECK-NEXT:  # %bb.32:
+; CHECK-NEXT:    mv s4, ra
+; CHECK-NEXT:  .LBB2_33:
+; CHECK-NEXT:    add t4, s9, s4
+; CHECK-NEXT:    sltu s9, t4, s9
+; CHECK-NEXT:    add t6, s8, s9
+; CHECK-NEXT:    add s5, s7, s11
+; CHECK-NEXT:    sltu s10, t6, s8
+; CHECK-NEXT:    add t1, s3, t1
+; CHECK-NEXT:    and s9, s9, s10
+; CHECK-NEXT:    beq s8, s2, .LBB2_35
+; CHECK-NEXT:  # %bb.34:
+; CHECK-NEXT:    sltu s6, s8, s2
+; CHECK-NEXT:  .LBB2_35:
+; CHECK-NEXT:    mul s11, a5, a5
+; CHECK-NEXT:    mulhu a5, a5, a5
+; CHECK-NEXT:    mulhu s8, a4, a4
+; CHECK-NEXT:    slli s10, t0, 1
+; CHECK-NEXT:    srli a7, a7, 63
+; CHECK-NEXT:    slli ra, t5, 1
+; CHECK-NEXT:    srli t0, t0, 63
+; CHECK-NEXT:    or t5, s10, a7
+; CHECK-NEXT:    or s10, ra, t0
+; CHECK-NEXT:    add s11, t5, s11
+; CHECK-NEXT:    add a7, s10, a5
+; CHECK-NEXT:    add s6, s11, s6
+; CHECK-NEXT:    sltu t0, s11, t5
+; CHECK-NEXT:    add a5, s6, s9
+; CHECK-NEXT:    add a7, a7, t0
+; CHECK-NEXT:    sltu t0, s6, s11
+; CHECK-NEXT:    sltu s6, a5, s6
+; CHECK-NEXT:    add a7, a7, t0
+; CHECK-NEXT:    sltu t0, s5, s7
+; CHECK-NEXT:    add a7, a7, s6
+; CHECK-NEXT:    slli s6, t1, 1
+; CHECK-NEXT:    srli s0, s0, 63
+; CHECK-NEXT:    beq a7, s10, .LBB2_37
+; CHECK-NEXT:  # %bb.36:
+; CHECK-NEXT:    sltu s7, a7, s10
+; CHECK-NEXT:    j .LBB2_38
+; CHECK-NEXT:  .LBB2_37:
+; CHECK-NEXT:    sltu s7, a5, t5
+; CHECK-NEXT:  .LBB2_38:
+; CHECK-NEXT:    or s0, s6, s0
+; CHECK-NEXT:    xor s9, a7, s10
+; CHECK-NEXT:    xor t5, a5, t5
+; CHECK-NEXT:    add s6, s8, t0
+; CHECK-NEXT:    or s10, t5, s9
+; CHECK-NEXT:    beq t6, s2, .LBB2_40
+; CHECK-NEXT:  # %bb.39:
+; CHECK-NEXT:    sltu s8, t6, s2
+; CHECK-NEXT:    add s6, s0, s6
+; CHECK-NEXT:    bnez s10, .LBB2_41
+; CHECK-NEXT:    j .LBB2_42
+; CHECK-NEXT:  .LBB2_40:
+; CHECK-NEXT:    sltu s8, t4, s1
+; CHECK-NEXT:    add s6, s0, s6
+; CHECK-NEXT:    beqz s10, .LBB2_42
+; CHECK-NEXT:  .LBB2_41:
+; CHECK-NEXT:    mv s8, s7
+; CHECK-NEXT:  .LBB2_42:
+; CHECK-NEXT:    xor s2, t6, s2
+; CHECK-NEXT:    xor s1, t4, s1
+; CHECK-NEXT:    or s2, s2, s9
+; CHECK-NEXT:    or t5, s1, t5
+; CHECK-NEXT:    or t5, t5, s2
+; CHECK-NEXT:    beqz t5, .LBB2_44
+; CHECK-NEXT:  # %bb.43:
+; CHECK-NEXT:    mv s4, s8
+; CHECK-NEXT:  .LBB2_44:
+; CHECK-NEXT:    add s4, s5, s4
+; CHECK-NEXT:    sltu s1, s4, s5
+; CHECK-NEXT:    add t5, s6, s1
+; CHECK-NEXT:    sltu s2, t5, s6
+; CHECK-NEXT:    and s1, s1, s2
+; CHECK-NEXT:    beq s6, s0, .LBB2_46
+; CHECK-NEXT:  # %bb.45:
+; CHECK-NEXT:    sltu t0, s6, s0
+; CHECK-NEXT:  .LBB2_46:
+; CHECK-NEXT:    mulhu a4, a4, a3
+; CHECK-NEXT:    mul s0, a3, a3
+; CHECK-NEXT:    mulhu a3, a3, a3
+; CHECK-NEXT:    sltu t3, s3, t3
+; CHECK-NEXT:    sltu s2, t1, s3
+; CHECK-NEXT:    add t3, t3, s2
+; CHECK-NEXT:    srli t1, t1, 63
+; CHECK-NEXT:    add a4, a4, t3
+; CHECK-NEXT:    mul a2, a2, a2
+; CHECK-NEXT:    slli t3, a4, 1
+; CHECK-NEXT:    srli a4, a4, 63
+; CHECK-NEXT:    or t1, t3, t1
+; CHECK-NEXT:    add a3, a4, a3
+; CHECK-NEXT:    add s0, t1, s0
+; CHECK-NEXT:    add t0, s0, t0
+; CHECK-NEXT:    sltu a4, s0, t1
+; CHECK-NEXT:    add s1, t0, s1
+; CHECK-NEXT:    add a3, a3, a4
+; CHECK-NEXT:    sltu a4, t0, s0
+; CHECK-NEXT:    sltu t0, s1, t0
+; CHECK-NEXT:    add a3, a3, a4
+; CHECK-NEXT:    sd a2, 0(a0)
+; CHECK-NEXT:    sd a1, 8(a0)
+; CHECK-NEXT:    sd a6, 16(a0)
+; CHECK-NEXT:    sd t2, 24(a0)
+; CHECK-NEXT:    add a3, a3, t0
+; CHECK-NEXT:    sd t4, 32(a0)
+; CHECK-NEXT:    sd t6, 40(a0)
+; CHECK-NEXT:    sd a5, 48(a0)
+; CHECK-NEXT:    sd a7, 56(a0)
+; CHECK-NEXT:    sd s4, 64(a0)
+; CHECK-NEXT:    sd t5, 72(a0)
+; CHECK-NEXT:    sd s1, 80(a0)
+; CHECK-NEXT:    sd a3, 88(a0)
+; CHECK-NEXT:    ld ra, 104(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s0, 96(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s1, 88(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s2, 80(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s3, 72(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s4, 64(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s5, 56(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s6, 48(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s7, 40(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s8, 32(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s9, 24(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s10, 16(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    ld s11, 8(sp) # 8-byte Folded Reload
+; CHECK-NEXT:    .cfi_restore ra
+; CHECK-NEXT:    .cfi_restore s0
+; CHECK-NEXT:    .cfi_restore s1
+; CHECK-NEXT:    .cfi_restore s2
+; CHECK-NEXT:    .cfi_restore s3
+; CHECK-NEXT:    .cfi_restore s4
+; CHECK-NEXT:    .cfi_restore s5
+; CHECK-NEXT:    .cfi_restore s6
+; CHECK-NEXT:    .cfi_restore s7
+; CHECK-NEXT:    .cfi_restore s8
+; CHECK-NEXT:    .cfi_restore s9
+; CHECK-NEXT:    .cfi_restore s10
+; CHECK-NEXT:    .cfi_restore s11
+; CHECK-NEXT:    addi sp, sp, 112
+; CHECK-NEXT:    .cfi_def_cfa_offset 0
+; CHECK-NEXT:    ret
+  %x = load i384, ptr %in
+  %z = zext i384 %x to i768
+  %r = mul i768 %z, %z
+  store i768 %r, ptr %out
+  ret void
+}
+
+; sign-extended operand: no known-zero limbs, truncated 4 x 4
+define void @sqr_i256_sext_i128(ptr %out, ptr %in) {
+; CHECK-LABEL: sqr_i256_sext_i128:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    ld a2, 8(a1)
+; CHECK-NEXT:    ld a1, 0(a1)
+; CHECK-NEXT:    srai a3, a2, 63
+; CHECK-NEXT:    mulhu a4, a1, a2
+; CHECK-NEXT:    mul a5, a1, a2
+; CHECK-NEXT:    mul a6, a1, a3
+; CHECK-NEXT:    mulhu a7, a1, a3
+; CHECK-NEXT:    mulhu t0, a1, a1
+; CHECK-NEXT:    mul a3, a2, a3
+; CHECK-NEXT:    mul t1, a2, a2
+; CHECK-NEXT:    mulhu a2, a2, a2
+; CHECK-NEXT:    mul a1, a1, a1
+; CHECK-NEXT:    srli t2, a5, 63
+; CHECK-NEXT:    slli a5, a5, 1
+; CHECK-NEXT:    add a4, a6, a4
+; CHECK-NEXT:    add a7, a6, a7
+; CHECK-NEXT:    add t0, a5, t0
+; CHECK-NEXT:    add a3, a7, a3
+; CHECK-NEXT:    slli a7, a4, 1
+; CHECK-NEXT:    sltu a6, a4, a6
+; CHECK-NEXT:    srli a4, a4, 63
+; CHECK-NEXT:    sltu a5, t0, a5
+; CHECK-NEXT:    or a7, a7, t2
+; CHECK-NEXT:    add a3, a3, a6
+; CHECK-NEXT:    add t1, a7, t1
+; CHECK-NEXT:    slli a3, a3, 1
+; CHECK-NEXT:    sltu a6, t1, a7
+; CHECK-NEXT:    or a3, a3, a4
+; CHECK-NEXT:    add a5, t1, a5
+; CHECK-NEXT:    add a2, a3, a2
+; CHECK-NEXT:    sltu a3, a5, t1
+; CHECK-NEXT:    add a2, a2, a6
+; CHECK-NEXT:    add a2, a2, a3
+; CHECK-NEXT:    sd a1, 0(a0)
+; CHECK-NEXT:    sd t0, 8(a0)
+; CHECK-NEXT:    sd a5, 16(a0)
+; CHECK-NEXT:    sd a2, 24(a0)
+; CHECK-NEXT:    ret
+  %x = load i128, ptr %in
+  %z = sext i128 %x to i256
+  %r = mul i256 %z, %z
+  store i256 %r, ptr %out
+  ret void
+}
+
+; a single limb: one product, not expanded here
+define void @sqr_i256_zext_i64(ptr %out, ptr %in) {
+; CHECK-LABEL: sqr_i256_zext_i64:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    ld a1, 0(a1)
+; CHECK-NEXT:    sd zero, 16(a0)
+; CHECK-NEXT:    mulhu a2, a1, a1
+; CHECK-NEXT:    mul a1, a1, a1
+; CHECK-NEXT:    sd a2, 8(a0)
+; CHECK-NEXT:    sd a1, 0(a0)
+; CHECK-NEXT:    sd zero, 24(a0)
+; CHECK-NEXT:    ret
+  %x = load i64, ptr %in
+  %z = zext i64 %x to i256
+  %r = mul i256 %z, %z
+  store i256 %r, ptr %out
+  ret void
+}
diff --git a/llvm/test/CodeGen/X86/dagcombine-cse.ll b/llvm/test/CodeGen/X86/dagcombine-cse.ll
index 3efd536adc4d18..b53a26ee24815e 100644
--- a/llvm/test/CodeGen/X86/dagcombine-cse.ll
+++ b/llvm/test/CodeGen/X86/dagcombine-cse.ll
@@ -50,54 +50,51 @@ define i96 @square_high(i96 %x) nounwind {
 ; X86-NEXT:    pushl %ebx
 ; X86-NEXT:    pushl %edi
 ; X86-NEXT:    pushl %esi
-; X86-NEXT:    pushl %eax
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
+; X86-NEXT:    subl $12, %esp
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebx
-; X86-NEXT:    movl %edi, %eax
-; X86-NEXT:    mull %edi
-; X86-NEXT:    movl %edx, %ecx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    movl %eax, %ebp
+; X86-NEXT:    mull %ecx
+; X86-NEXT:    movl %edx, %edi
+; X86-NEXT:    movl %eax, (%esp) ## 4-byte Spill
 ; X86-NEXT:    movl %ebx, %eax
-; X86-NEXT:    mull %edi
-; X86-NEXT:    addl %eax, %ecx
-; X86-NEXT:    movl %edx, %ebp
-; X86-NEXT:    adcl $0, %ebp
-; X86-NEXT:    addl %eax, %ecx
-; X86-NEXT:    adcl %edx, %ebp
-; X86-NEXT:    setb %al
-; X86-NEXT:    movzbl %al, %ecx
+; X86-NEXT:    mull %ecx
+; X86-NEXT:    movl %edx, %esi
+; X86-NEXT:    movl %eax, %ecx
 ; X86-NEXT:    movl %ebx, %eax
-; X86-NEXT:    mull %ebx
-; X86-NEXT:    movl %eax, %ebx
-; X86-NEXT:    addl %ebp, %ebx
-; X86-NEXT:    adcl %edx, %ecx
-; X86-NEXT:    movl %esi, %eax
-; X86-NEXT:    mull %edi
-; X86-NEXT:    movl %edx, (%esp) ## 4-byte Spill
+; X86-NEXT:    mull %ebp
 ; X86-NEXT:    movl %eax, %ebp
-; X86-NEXT:    movl %esi, %eax
-; X86-NEXT:    mull {{[0-9]+}}(%esp)
-; X86-NEXT:    movl %edx, %esi
-; X86-NEXT:    movl %eax, %edi
-; X86-NEXT:    addl (%esp), %edi ## 4-byte Folded Reload
-; X86-NEXT:    adcl $0, %esi
-; X86-NEXT:    addl %ebp, %ebx
-; X86-NEXT:    adcl %edi, %ecx
-; X86-NEXT:    movl %esi, %eax
-; X86-NEXT:    adcl $0, %eax
-; X86-NEXT:    setb %dl
-; X86-NEXT:    addl %ebp, %ebx
-; X86-NEXT:    adcl %ecx, %edi
-; X86-NEXT:    movzbl %dl, %ecx
-; X86-NEXT:    adcl %eax, %esi
-; X86-NEXT:    adcl $0, %ecx
+; X86-NEXT:    movl %edx, %ebx
+; X86-NEXT:    addl %ecx, %ebx
+; X86-NEXT:    adcl (%esp), %esi ## 4-byte Folded Reload
+; X86-NEXT:    adcl $0, %edi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    mull %eax
+; X86-NEXT:    movl %edx, (%esp) ## 4-byte Spill
+; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) ## 4-byte Spill
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    mull %eax
+; X86-NEXT:    movl %edx, %ecx
+; X86-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) ## 4-byte Spill
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    mull %eax
-; X86-NEXT:    addl %eax, %esi
-; X86-NEXT:    adcl %edx, %ecx
 ; X86-NEXT:    movl %edi, %eax
-; X86-NEXT:    movl %esi, %edx
-; X86-NEXT:    addl $4, %esp
+; X86-NEXT:    shldl $1, %esi, %eax
+; X86-NEXT:    shrl $31, %edi
+; X86-NEXT:    shldl $1, %ebx, %esi
+; X86-NEXT:    shldl $1, %ebp, %ebx
+; X86-NEXT:    addl %ebp, %ebp
+; X86-NEXT:    addl %edx, %ebp
+; X86-NEXT:    adcl %ebx, {{[-0-9]+}}(%e{{[sb]}}p) ## 4-byte Folded Spill
+; X86-NEXT:    adcl %esi, %ecx
+; X86-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %edx ## 4-byte Reload
+; X86-NEXT:    adcl %eax, %edx
+; X86-NEXT:    movl (%esp), %esi ## 4-byte Reload
+; X86-NEXT:    adcl %edi, %esi
+; X86-NEXT:    movl %ecx, %eax
+; X86-NEXT:    movl %esi, %ecx
+; X86-NEXT:    addl $12, %esp
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    popl %edi
 ; X86-NEXT:    popl %ebx
@@ -107,19 +104,17 @@ define i96 @square_high(i96 %x) nounwind {
 ; X64-LABEL: square_high:
 ; X64:       ## %bb.0: ## %entry
 ; X64-NEXT:    movl %esi, %ecx
-; X64-NEXT:    movq %rcx, %rax
-; X64-NEXT:    mulq %rdi
+; X64-NEXT:    movq %rdi, %rax
+; X64-NEXT:    mulq %rcx
 ; X64-NEXT:    movq %rdx, %rsi
 ; X64-NEXT:    movq %rax, %r8
+; X64-NEXT:    shldq $1, %rax, %rsi
+; X64-NEXT:    addq %rax, %r8
+; X64-NEXT:    imulq %rcx, %rcx
 ; X64-NEXT:    movq %rdi, %rax
 ; X64-NEXT:    mulq %rdi
-; X64-NEXT:    addq %r8, %rdx
-; X64-NEXT:    movq %rsi, %rax
-; X64-NEXT:    adcq $0, %rax
 ; X64-NEXT:    addq %rdx, %r8
-; X64-NEXT:    adcq %rsi, %rax
-; X64-NEXT:    imulq %rcx, %rcx
-; X64-NEXT:    addq %rax, %rcx
+; X64-NEXT:    adcq %rsi, %rcx
 ; X64-NEXT:    shrdq $32, %rcx, %r8
 ; X64-NEXT:    shrq $32, %rcx
 ; X64-NEXT:    movq %r8, %rax
diff --git a/llvm/test/CodeGen/X86/wide-int-square-i512.ll b/llvm/test/CodeGen/X86/wide-int-square-i512.ll
new file mode 100644
index 00000000000000..36d12de3e68f40
--- /dev/null
+++ b/llvm/test/CodeGen/X86/wide-int-square-i512.ll
@@ -0,0 +1,302 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
+; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu | FileCheck %s --check-prefix=X64
+; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=+bmi2 | FileCheck %s --check-prefix=BMI2
+
+; truncated square, 8 x 8 limbs: the products whose high limb falls beyond
+; the result are computed with a plain mul
+define void @sqr_i512(ptr %out, ptr %in) {
+; X64-LABEL: sqr_i512:
+; X64:       # %bb.0:
+; X64-NEXT:    pushq %rbp
+; X64-NEXT:    .cfi_def_cfa_offset 16
+; X64-NEXT:    pushq %r15
+; X64-NEXT:    .cfi_def_cfa_offset 24
+; X64-NEXT:    pushq %r14
+; X64-NEXT:    .cfi_def_cfa_offset 32
+; X64-NEXT:    pushq %r13
+; X64-NEXT:    .cfi_def_cfa_offset 40
+; X64-NEXT:    pushq %r12
+; X64-NEXT:    .cfi_def_cfa_offset 48
+; X64-NEXT:    pushq %rbx
+; X64-NEXT:    .cfi_def_cfa_offset 56
+; X64-NEXT:    .cfi_offset %rbx, -56
+; X64-NEXT:    .cfi_offset %r12, -48
+; X64-NEXT:    .cfi_offset %r13, -40
+; X64-NEXT:    .cfi_offset %r14, -32
+; X64-NEXT:    .cfi_offset %r15, -24
+; X64-NEXT:    .cfi_offset %rbp, -16
+; X64-NEXT:    movq %rdi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq 24(%rsi), %r15
+; X64-NEXT:    movq 16(%rsi), %r11
+; X64-NEXT:    movq 32(%rsi), %rbx
+; X64-NEXT:    movq 40(%rsi), %r13
+; X64-NEXT:    movq (%rsi), %rcx
+; X64-NEXT:    movq 8(%rsi), %r14
+; X64-NEXT:    movq 48(%rsi), %r10
+; X64-NEXT:    movq %rcx, %rax
+; X64-NEXT:    mulq %r10
+; X64-NEXT:    movq %rax, %rdi
+; X64-NEXT:    movq 56(%rsi), %r8
+; X64-NEXT:    imulq %rcx, %r8
+; X64-NEXT:    addq %rdx, %r8
+; X64-NEXT:    imulq %r14, %r10
+; X64-NEXT:    movq %rcx, %rax
+; X64-NEXT:    mulq %r13
+; X64-NEXT:    movq %rax, %rsi
+; X64-NEXT:    movq %rdx, %r9
+; X64-NEXT:    addq %rdi, %r9
+; X64-NEXT:    adcq %r8, %r10
+; X64-NEXT:    movq %r14, %rax
+; X64-NEXT:    mulq %r13
+; X64-NEXT:    movq %rdx, %r8
+; X64-NEXT:    movq %rax, %rdi
+; X64-NEXT:    movq %rcx, %rax
+; X64-NEXT:    mulq %rbx
+; X64-NEXT:    movq %rax, %rbp
+; X64-NEXT:    movq %rdx, %r12
+; X64-NEXT:    addq %rsi, %r12
+; X64-NEXT:    adcq %r9, %rdi
+; X64-NEXT:    adcq %r10, %r8
+; X64-NEXT:    movq %r11, %r9
+; X64-NEXT:    imulq %r11, %r13
+; X64-NEXT:    movq %r14, %rax
+; X64-NEXT:    movq %r14, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    mulq %rbx
+; X64-NEXT:    movq %rbx, %rsi
+; X64-NEXT:    movq %rdx, %r10
+; X64-NEXT:    movq %rax, %r11
+; X64-NEXT:    movq %rcx, %rax
+; X64-NEXT:    mulq %r15
+; X64-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq %rdx, %rbx
+; X64-NEXT:    addq %rbp, %rbx
+; X64-NEXT:    adcq %r12, %r11
+; X64-NEXT:    adcq %rdi, %r10
+; X64-NEXT:    adcq %r8, %r13
+; X64-NEXT:    movq %r9, %rax
+; X64-NEXT:    movq %r9, %rdi
+; X64-NEXT:    mulq %rsi
+; X64-NEXT:    movq %rdx, %r12
+; X64-NEXT:    movq %rax, %rbp
+; X64-NEXT:    movq %r14, %rax
+; X64-NEXT:    movq %r15, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    mulq %r15
+; X64-NEXT:    movq %rdx, %r9
+; X64-NEXT:    movq %rax, %r8
+; X64-NEXT:    movq %rcx, %rax
+; X64-NEXT:    mulq %rdi
+; X64-NEXT:    movq %rdi, %r14
+; X64-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq %rdx, %rdi
+; X64-NEXT:    addq {{[-0-9]+}}(%r{{[sb]}}p), %rdi # 8-byte Folded Reload
+; X64-NEXT:    adcq %rbx, %r8
+; X64-NEXT:    adcq %r11, %r9
+; X64-NEXT:    adcq %r10, %rbp
+; X64-NEXT:    adcq %r13, %r12
+; X64-NEXT:    imulq %r15, %rsi
+; X64-NEXT:    movq %rsi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq %r14, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq %r14, %rax
+; X64-NEXT:    mulq %r15
+; X64-NEXT:    movq %rdx, %r10
+; X64-NEXT:    movq %rax, %r11
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rsi # 8-byte Reload
+; X64-NEXT:    movq %rsi, %rax
+; X64-NEXT:    mulq %r14
+; X64-NEXT:    movq %rdx, %r14
+; X64-NEXT:    movq %rax, %r15
+; X64-NEXT:    movq %rcx, %rax
+; X64-NEXT:    mulq %rsi
+; X64-NEXT:    movq %rax, %rbx
+; X64-NEXT:    movq %rdx, %r13
+; X64-NEXT:    addq {{[-0-9]+}}(%r{{[sb]}}p), %r13 # 8-byte Folded Reload
+; X64-NEXT:    adcq %rdi, %r15
+; X64-NEXT:    adcq %r8, %r14
+; X64-NEXT:    adcq %r9, %r11
+; X64-NEXT:    adcq %rbp, %r10
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; X64-NEXT:    adcq %r12, %rax
+; X64-NEXT:    shldq $1, %r10, %rax
+; X64-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    shldq $1, %r11, %r10
+; X64-NEXT:    shldq $1, %r14, %r11
+; X64-NEXT:    shldq $1, %r15, %r14
+; X64-NEXT:    shldq $1, %r13, %r15
+; X64-NEXT:    shldq $1, %rbx, %r13
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; X64-NEXT:    mulq %rax
+; X64-NEXT:    movq %rdx, %r9
+; X64-NEXT:    movq %rax, %r8
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; X64-NEXT:    mulq %rax
+; X64-NEXT:    movq %rdx, %rdi
+; X64-NEXT:    movq %rax, %r12
+; X64-NEXT:    movq %rsi, %rax
+; X64-NEXT:    mulq %rsi
+; X64-NEXT:    movq %rdx, %rsi
+; X64-NEXT:    movq %rax, %rbp
+; X64-NEXT:    movq %rcx, %rax
+; X64-NEXT:    mulq %rcx
+; X64-NEXT:    addq %rbx, %rbx
+; X64-NEXT:    addq %rdx, %rbx
+; X64-NEXT:    adcq %r13, %rbp
+; X64-NEXT:    adcq %r15, %rsi
+; X64-NEXT:    adcq %r14, %r12
+; X64-NEXT:    adcq %r11, %rdi
+; X64-NEXT:    adcq %r10, %r8
+; X64-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Folded Reload
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rcx # 8-byte Reload
+; X64-NEXT:    movq %rax, (%rcx)
+; X64-NEXT:    movq %rbx, 8(%rcx)
+; X64-NEXT:    movq %rbp, 16(%rcx)
+; X64-NEXT:    movq %rsi, 24(%rcx)
+; X64-NEXT:    movq %r12, 32(%rcx)
+; X64-NEXT:    movq %rdi, 40(%rcx)
+; X64-NEXT:    movq %r8, 48(%rcx)
+; X64-NEXT:    movq %r9, 56(%rcx)
+; X64-NEXT:    popq %rbx
+; X64-NEXT:    .cfi_def_cfa_offset 48
+; X64-NEXT:    popq %r12
+; X64-NEXT:    .cfi_def_cfa_offset 40
+; X64-NEXT:    popq %r13
+; X64-NEXT:    .cfi_def_cfa_offset 32
+; X64-NEXT:    popq %r14
+; X64-NEXT:    .cfi_def_cfa_offset 24
+; X64-NEXT:    popq %r15
+; X64-NEXT:    .cfi_def_cfa_offset 16
+; X64-NEXT:    popq %rbp
+; X64-NEXT:    .cfi_def_cfa_offset 8
+; X64-NEXT:    retq
+;
+; BMI2-LABEL: sqr_i512:
+; BMI2:       # %bb.0:
+; BMI2-NEXT:    pushq %rbp
+; BMI2-NEXT:    .cfi_def_cfa_offset 16
+; BMI2-NEXT:    pushq %r15
+; BMI2-NEXT:    .cfi_def_cfa_offset 24
+; BMI2-NEXT:    pushq %r14
+; BMI2-NEXT:    .cfi_def_cfa_offset 32
+; BMI2-NEXT:    pushq %r13
+; BMI2-NEXT:    .cfi_def_cfa_offset 40
+; BMI2-NEXT:    pushq %r12
+; BMI2-NEXT:    .cfi_def_cfa_offset 48
+; BMI2-NEXT:    pushq %rbx
+; BMI2-NEXT:    .cfi_def_cfa_offset 56
+; BMI2-NEXT:    .cfi_offset %rbx, -56
+; BMI2-NEXT:    .cfi_offset %r12, -48
+; BMI2-NEXT:    .cfi_offset %r13, -40
+; BMI2-NEXT:    .cfi_offset %r14, -32
+; BMI2-NEXT:    .cfi_offset %r15, -24
+; BMI2-NEXT:    .cfi_offset %rbp, -16
+; BMI2-NEXT:    movq %rdi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq 24(%rsi), %rcx
+; BMI2-NEXT:    movq 16(%rsi), %r9
+; BMI2-NEXT:    movq 32(%rsi), %r10
+; BMI2-NEXT:    movq 40(%rsi), %r11
+; BMI2-NEXT:    movq (%rsi), %rax
+; BMI2-NEXT:    movq 8(%rsi), %r8
+; BMI2-NEXT:    movq 48(%rsi), %rbx
+; BMI2-NEXT:    movq %rax, %rdx
+; BMI2-NEXT:    mulxq %rbx, %r14, %rdx
+; BMI2-NEXT:    movq 56(%rsi), %rsi
+; BMI2-NEXT:    imulq %rax, %rsi
+; BMI2-NEXT:    addq %rdx, %rsi
+; BMI2-NEXT:    imulq %r8, %rbx
+; BMI2-NEXT:    movq %rax, %rdx
+; BMI2-NEXT:    mulxq %r11, %r15, %r12
+; BMI2-NEXT:    addq %r14, %r12
+; BMI2-NEXT:    adcq %rsi, %rbx
+; BMI2-NEXT:    movq %r8, %rdx
+; BMI2-NEXT:    mulxq %r11, %rsi, %r14
+; BMI2-NEXT:    movq %rax, %rdx
+; BMI2-NEXT:    mulxq %r10, %r13, %rbp
+; BMI2-NEXT:    addq %r15, %rbp
+; BMI2-NEXT:    adcq %r12, %rsi
+; BMI2-NEXT:    adcq %rbx, %r14
+; BMI2-NEXT:    imulq %r9, %r11
+; BMI2-NEXT:    movq %r8, %rdx
+; BMI2-NEXT:    mulxq %r10, %rbx, %r15
+; BMI2-NEXT:    movq %rax, %rdx
+; BMI2-NEXT:    mulxq %rcx, %rdx, %r12
+; BMI2-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    addq %r13, %r12
+; BMI2-NEXT:    adcq %rbp, %rbx
+; BMI2-NEXT:    adcq %rsi, %r15
+; BMI2-NEXT:    adcq %r14, %r11
+; BMI2-NEXT:    movq %r9, %rdx
+; BMI2-NEXT:    mulxq %r10, %rbp, %r14
+; BMI2-NEXT:    movq %r8, %rdx
+; BMI2-NEXT:    movq %rcx, %rsi
+; BMI2-NEXT:    mulxq %rcx, %rcx, %rdi
+; BMI2-NEXT:    movq %rax, %rdx
+; BMI2-NEXT:    mulxq %r9, %rdx, %r13
+; BMI2-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    addq {{[-0-9]+}}(%r{{[sb]}}p), %r13 # 8-byte Folded Reload
+; BMI2-NEXT:    adcq %r12, %rcx
+; BMI2-NEXT:    adcq %rbx, %rdi
+; BMI2-NEXT:    adcq %r15, %rbp
+; BMI2-NEXT:    adcq %r11, %r14
+; BMI2-NEXT:    movq %r14, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq %rsi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    imulq %rsi, %r10
+; BMI2-NEXT:    movq %r9, %rdx
+; BMI2-NEXT:    mulxq %rsi, %rbx, %rsi
+; BMI2-NEXT:    movq %r8, %rdx
+; BMI2-NEXT:    mulxq %r9, %r15, %r14
+; BMI2-NEXT:    movq %rax, %rdx
+; BMI2-NEXT:    mulxq %r8, %r11, %r12
+; BMI2-NEXT:    addq {{[-0-9]+}}(%r{{[sb]}}p), %r12 # 8-byte Folded Reload
+; BMI2-NEXT:    adcq %r13, %r15
+; BMI2-NEXT:    adcq %rcx, %r14
+; BMI2-NEXT:    adcq %rdi, %rbx
+; BMI2-NEXT:    adcq %rbp, %rsi
+; BMI2-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %r10 # 8-byte Folded Reload
+; BMI2-NEXT:    shldq $1, %rsi, %r10
+; BMI2-NEXT:    shldq $1, %rbx, %rsi
+; BMI2-NEXT:    shldq $1, %r14, %rbx
+; BMI2-NEXT:    shldq $1, %r15, %r14
+; BMI2-NEXT:    shldq $1, %r12, %r15
+; BMI2-NEXT:    shldq $1, %r11, %r12
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; BMI2-NEXT:    mulxq %rdx, %rcx, %rdi
+; BMI2-NEXT:    movq %r9, %rdx
+; BMI2-NEXT:    mulxq %r9, %r9, %r13
+; BMI2-NEXT:    movq %r8, %rdx
+; BMI2-NEXT:    mulxq %r8, %r8, %rbp
+; BMI2-NEXT:    movq %rax, %rdx
+; BMI2-NEXT:    mulxq %rax, %rdx, %rax
+; BMI2-NEXT:    addq %r11, %r11
+; BMI2-NEXT:    addq %rax, %r11
+; BMI2-NEXT:    adcq %r12, %r8
+; BMI2-NEXT:    adcq %r15, %rbp
+; BMI2-NEXT:    adcq %r14, %r9
+; BMI2-NEXT:    adcq %rbx, %r13
+; BMI2-NEXT:    adcq %rsi, %rcx
+; BMI2-NEXT:    adcq %r10, %rdi
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; BMI2-NEXT:    movq %rdx, (%rax)
+; BMI2-NEXT:    movq %r11, 8(%rax)
+; BMI2-NEXT:    movq %r8, 16(%rax)
+; BMI2-NEXT:    movq %rbp, 24(%rax)
+; BMI2-NEXT:    movq %r9, 32(%rax)
+; BMI2-NEXT:    movq %r13, 40(%rax)
+; BMI2-NEXT:    movq %rcx, 48(%rax)
+; BMI2-NEXT:    movq %rdi, 56(%rax)
+; BMI2-NEXT:    popq %rbx
+; BMI2-NEXT:    .cfi_def_cfa_offset 48
+; BMI2-NEXT:    popq %r12
+; BMI2-NEXT:    .cfi_def_cfa_offset 40
+; BMI2-NEXT:    popq %r13
+; BMI2-NEXT:    .cfi_def_cfa_offset 32
+; BMI2-NEXT:    popq %r14
+; BMI2-NEXT:    .cfi_def_cfa_offset 24
+; BMI2-NEXT:    popq %r15
+; BMI2-NEXT:    .cfi_def_cfa_offset 16
+; BMI2-NEXT:    popq %rbp
+; BMI2-NEXT:    .cfi_def_cfa_offset 8
+; BMI2-NEXT:    retq
+  %x = load i512, ptr %in
+  %r = mul i512 %x, %x
+  store i512 %r, ptr %out
+  ret void
+}
diff --git a/llvm/test/CodeGen/X86/wide-int-square.ll b/llvm/test/CodeGen/X86/wide-int-square.ll
new file mode 100644
index 00000000000000..a3d7b183dd9d5a
--- /dev/null
+++ b/llvm/test/CodeGen/X86/wide-int-square.ll
@@ -0,0 +1,809 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
+; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu | FileCheck %s --check-prefix=X64
+; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=+bmi2 | FileCheck %s --check-prefix=BMI2
+
+; truncated square, 4 x 4 limbs
+define void @sqr_i256(ptr %out, ptr %in) {
+; X64-LABEL: sqr_i256:
+; X64:       # %bb.0:
+; X64-NEXT:    pushq %rbx
+; X64-NEXT:    .cfi_def_cfa_offset 16
+; X64-NEXT:    .cfi_offset %rbx, -16
+; X64-NEXT:    movq (%rsi), %rcx
+; X64-NEXT:    movq 8(%rsi), %r8
+; X64-NEXT:    movq 16(%rsi), %r11
+; X64-NEXT:    movq %rcx, %rax
+; X64-NEXT:    mulq %r11
+; X64-NEXT:    movq %rax, %r9
+; X64-NEXT:    movq 24(%rsi), %rbx
+; X64-NEXT:    imulq %rcx, %rbx
+; X64-NEXT:    addq %rdx, %rbx
+; X64-NEXT:    imulq %r8, %r11
+; X64-NEXT:    movq %rcx, %rax
+; X64-NEXT:    mulq %r8
+; X64-NEXT:    movq %rax, %rsi
+; X64-NEXT:    movq %rdx, %r10
+; X64-NEXT:    addq %r9, %r10
+; X64-NEXT:    adcq %rbx, %r11
+; X64-NEXT:    shldq $1, %r10, %r11
+; X64-NEXT:    shldq $1, %rax, %r10
+; X64-NEXT:    movq %r8, %rax
+; X64-NEXT:    mulq %r8
+; X64-NEXT:    movq %rdx, %r8
+; X64-NEXT:    movq %rax, %r9
+; X64-NEXT:    movq %rcx, %rax
+; X64-NEXT:    mulq %rcx
+; X64-NEXT:    addq %rsi, %rsi
+; X64-NEXT:    addq %rdx, %rsi
+; X64-NEXT:    adcq %r10, %r9
+; X64-NEXT:    adcq %r11, %r8
+; X64-NEXT:    movq %rax, (%rdi)
+; X64-NEXT:    movq %rsi, 8(%rdi)
+; X64-NEXT:    movq %r9, 16(%rdi)
+; X64-NEXT:    movq %r8, 24(%rdi)
+; X64-NEXT:    popq %rbx
+; X64-NEXT:    .cfi_def_cfa_offset 8
+; X64-NEXT:    retq
+;
+; BMI2-LABEL: sqr_i256:
+; BMI2:       # %bb.0:
+; BMI2-NEXT:    movq (%rsi), %rax
+; BMI2-NEXT:    movq 8(%rsi), %rcx
+; BMI2-NEXT:    movq 16(%rsi), %r8
+; BMI2-NEXT:    movq %rax, %rdx
+; BMI2-NEXT:    mulxq %r8, %r9, %rdx
+; BMI2-NEXT:    movq 24(%rsi), %rsi
+; BMI2-NEXT:    imulq %rax, %rsi
+; BMI2-NEXT:    addq %rdx, %rsi
+; BMI2-NEXT:    imulq %rcx, %r8
+; BMI2-NEXT:    movq %rax, %rdx
+; BMI2-NEXT:    mulxq %rcx, %r11, %r10
+; BMI2-NEXT:    addq %r9, %r10
+; BMI2-NEXT:    adcq %rsi, %r8
+; BMI2-NEXT:    shldq $1, %r10, %r8
+; BMI2-NEXT:    shldq $1, %r11, %r10
+; BMI2-NEXT:    movq %rcx, %rdx
+; BMI2-NEXT:    mulxq %rcx, %rcx, %rsi
+; BMI2-NEXT:    movq %rax, %rdx
+; BMI2-NEXT:    mulxq %rax, %rdx, %rax
+; BMI2-NEXT:    addq %r11, %r11
+; BMI2-NEXT:    addq %rax, %r11
+; BMI2-NEXT:    adcq %r10, %rcx
+; BMI2-NEXT:    adcq %r8, %rsi
+; BMI2-NEXT:    movq %rdx, (%rdi)
+; BMI2-NEXT:    movq %r11, 8(%rdi)
+; BMI2-NEXT:    movq %rcx, 16(%rdi)
+; BMI2-NEXT:    movq %rsi, 24(%rdi)
+; BMI2-NEXT:    retq
+  %x = load i256, ptr %in
+  %r = mul i256 %x, %x
+  store i256 %r, ptr %out
+  ret void
+}
+
+; full square, 4 limbs -> 8 limbs
+define void @sqr_i512_zext_i256(ptr %out, ptr %in) {
+; X64-LABEL: sqr_i512_zext_i256:
+; X64:       # %bb.0:
+; X64-NEXT:    pushq %rbp
+; X64-NEXT:    .cfi_def_cfa_offset 16
+; X64-NEXT:    pushq %r15
+; X64-NEXT:    .cfi_def_cfa_offset 24
+; X64-NEXT:    pushq %r14
+; X64-NEXT:    .cfi_def_cfa_offset 32
+; X64-NEXT:    pushq %r13
+; X64-NEXT:    .cfi_def_cfa_offset 40
+; X64-NEXT:    pushq %r12
+; X64-NEXT:    .cfi_def_cfa_offset 48
+; X64-NEXT:    pushq %rbx
+; X64-NEXT:    .cfi_def_cfa_offset 56
+; X64-NEXT:    .cfi_offset %rbx, -56
+; X64-NEXT:    .cfi_offset %r12, -48
+; X64-NEXT:    .cfi_offset %r13, -40
+; X64-NEXT:    .cfi_offset %r14, -32
+; X64-NEXT:    .cfi_offset %r15, -24
+; X64-NEXT:    .cfi_offset %rbp, -16
+; X64-NEXT:    movq 16(%rsi), %r9
+; X64-NEXT:    movq (%rsi), %r12
+; X64-NEXT:    movq 8(%rsi), %rax
+; X64-NEXT:    movq 24(%rsi), %rbx
+; X64-NEXT:    movq %rax, %r11
+; X64-NEXT:    mulq %rbx
+; X64-NEXT:    movq %rdx, %r13
+; X64-NEXT:    movq %rax, %rsi
+; X64-NEXT:    movq %r12, %rax
+; X64-NEXT:    mulq %r9
+; X64-NEXT:    movq %rdx, %r8
+; X64-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq %r12, %rax
+; X64-NEXT:    movq %r12, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    mulq %rbx
+; X64-NEXT:    movq %rdx, %rbp
+; X64-NEXT:    movq %rax, %rcx
+; X64-NEXT:    addq %r8, %rcx
+; X64-NEXT:    adcq %rsi, %rbp
+; X64-NEXT:    adcq $0, %r13
+; X64-NEXT:    movq %r9, %rax
+; X64-NEXT:    mulq %rbx
+; X64-NEXT:    movq %rdx, %r8
+; X64-NEXT:    movq %rax, %r10
+; X64-NEXT:    movq %r11, %rsi
+; X64-NEXT:    movq %r11, %rax
+; X64-NEXT:    mulq %r9
+; X64-NEXT:    movq %rdx, %r14
+; X64-NEXT:    movq %rax, %r15
+; X64-NEXT:    movq %r12, %rax
+; X64-NEXT:    mulq %r11
+; X64-NEXT:    movq %rax, %r11
+; X64-NEXT:    movq %rdx, %r12
+; X64-NEXT:    addq {{[-0-9]+}}(%r{{[sb]}}p), %r12 # 8-byte Folded Reload
+; X64-NEXT:    adcq %rcx, %r15
+; X64-NEXT:    adcq %rbp, %r14
+; X64-NEXT:    adcq %r13, %r10
+; X64-NEXT:    adcq $0, %r8
+; X64-NEXT:    movq %r8, %rax
+; X64-NEXT:    shldq $1, %r10, %rax
+; X64-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    shldq $1, %r14, %r10
+; X64-NEXT:    shrq $63, %r8
+; X64-NEXT:    shldq $1, %r15, %r14
+; X64-NEXT:    shldq $1, %r12, %r15
+; X64-NEXT:    shldq $1, %r11, %r12
+; X64-NEXT:    movq %rbx, %rax
+; X64-NEXT:    mulq %rbx
+; X64-NEXT:    movq %rdx, %rbx
+; X64-NEXT:    movq %rax, %r13
+; X64-NEXT:    movq %r9, %rax
+; X64-NEXT:    mulq %r9
+; X64-NEXT:    movq %rdx, %r9
+; X64-NEXT:    movq %rax, %rbp
+; X64-NEXT:    movq %rsi, %rax
+; X64-NEXT:    mulq %rax
+; X64-NEXT:    movq %rdx, %rcx
+; X64-NEXT:    movq %rax, %rsi
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; X64-NEXT:    mulq %rax
+; X64-NEXT:    addq %r11, %r11
+; X64-NEXT:    addq %rdx, %r11
+; X64-NEXT:    adcq %r12, %rsi
+; X64-NEXT:    adcq %r15, %rcx
+; X64-NEXT:    adcq %r14, %rbp
+; X64-NEXT:    adcq %r10, %r9
+; X64-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %r13 # 8-byte Folded Reload
+; X64-NEXT:    adcq %r8, %rbx
+; X64-NEXT:    movq %rax, (%rdi)
+; X64-NEXT:    movq %r11, 8(%rdi)
+; X64-NEXT:    movq %rsi, 16(%rdi)
+; X64-NEXT:    movq %rcx, 24(%rdi)
+; X64-NEXT:    movq %rbp, 32(%rdi)
+; X64-NEXT:    movq %r9, 40(%rdi)
+; X64-NEXT:    movq %r13, 48(%rdi)
+; X64-NEXT:    movq %rbx, 56(%rdi)
+; X64-NEXT:    popq %rbx
+; X64-NEXT:    .cfi_def_cfa_offset 48
+; X64-NEXT:    popq %r12
+; X64-NEXT:    .cfi_def_cfa_offset 40
+; X64-NEXT:    popq %r13
+; X64-NEXT:    .cfi_def_cfa_offset 32
+; X64-NEXT:    popq %r14
+; X64-NEXT:    .cfi_def_cfa_offset 24
+; X64-NEXT:    popq %r15
+; X64-NEXT:    .cfi_def_cfa_offset 16
+; X64-NEXT:    popq %rbp
+; X64-NEXT:    .cfi_def_cfa_offset 8
+; X64-NEXT:    retq
+;
+; BMI2-LABEL: sqr_i512_zext_i256:
+; BMI2:       # %bb.0:
+; BMI2-NEXT:    pushq %rbp
+; BMI2-NEXT:    .cfi_def_cfa_offset 16
+; BMI2-NEXT:    pushq %r15
+; BMI2-NEXT:    .cfi_def_cfa_offset 24
+; BMI2-NEXT:    pushq %r14
+; BMI2-NEXT:    .cfi_def_cfa_offset 32
+; BMI2-NEXT:    pushq %r13
+; BMI2-NEXT:    .cfi_def_cfa_offset 40
+; BMI2-NEXT:    pushq %r12
+; BMI2-NEXT:    .cfi_def_cfa_offset 48
+; BMI2-NEXT:    pushq %rbx
+; BMI2-NEXT:    .cfi_def_cfa_offset 56
+; BMI2-NEXT:    .cfi_offset %rbx, -56
+; BMI2-NEXT:    .cfi_offset %r12, -48
+; BMI2-NEXT:    .cfi_offset %r13, -40
+; BMI2-NEXT:    .cfi_offset %r14, -32
+; BMI2-NEXT:    .cfi_offset %r15, -24
+; BMI2-NEXT:    .cfi_offset %rbp, -16
+; BMI2-NEXT:    movq 16(%rsi), %rcx
+; BMI2-NEXT:    movq (%rsi), %rax
+; BMI2-NEXT:    movq 8(%rsi), %r9
+; BMI2-NEXT:    movq 24(%rsi), %r8
+; BMI2-NEXT:    movq %r9, %rdx
+; BMI2-NEXT:    mulxq %r8, %rsi, %r12
+; BMI2-NEXT:    movq %rax, %rdx
+; BMI2-NEXT:    mulxq %rcx, %rdx, %r10
+; BMI2-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq %rax, %rdx
+; BMI2-NEXT:    mulxq %r8, %rbp, %r13
+; BMI2-NEXT:    addq %r10, %rbp
+; BMI2-NEXT:    adcq %rsi, %r13
+; BMI2-NEXT:    adcq $0, %r12
+; BMI2-NEXT:    movq %rcx, %rdx
+; BMI2-NEXT:    mulxq %r8, %r11, %rsi
+; BMI2-NEXT:    movq %r9, %rdx
+; BMI2-NEXT:    mulxq %rcx, %r14, %rbx
+; BMI2-NEXT:    movq %rax, %rdx
+; BMI2-NEXT:    mulxq %r9, %r10, %r15
+; BMI2-NEXT:    addq {{[-0-9]+}}(%r{{[sb]}}p), %r15 # 8-byte Folded Reload
+; BMI2-NEXT:    adcq %rbp, %r14
+; BMI2-NEXT:    adcq %r13, %rbx
+; BMI2-NEXT:    adcq %r12, %r11
+; BMI2-NEXT:    adcq $0, %rsi
+; BMI2-NEXT:    movq %rsi, %rdx
+; BMI2-NEXT:    shldq $1, %r11, %rdx
+; BMI2-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    shldq $1, %rbx, %r11
+; BMI2-NEXT:    shrq $63, %rsi
+; BMI2-NEXT:    shldq $1, %r14, %rbx
+; BMI2-NEXT:    shldq $1, %r15, %r14
+; BMI2-NEXT:    shldq $1, %r10, %r15
+; BMI2-NEXT:    movq %r8, %rdx
+; BMI2-NEXT:    mulxq %r8, %r8, %r13
+; BMI2-NEXT:    movq %rcx, %rdx
+; BMI2-NEXT:    mulxq %rcx, %rcx, %rbp
+; BMI2-NEXT:    movq %r9, %rdx
+; BMI2-NEXT:    mulxq %r9, %r9, %r12
+; BMI2-NEXT:    movq %rax, %rdx
+; BMI2-NEXT:    mulxq %rax, %rdx, %rax
+; BMI2-NEXT:    addq %r10, %r10
+; BMI2-NEXT:    addq %rax, %r10
+; BMI2-NEXT:    adcq %r15, %r9
+; BMI2-NEXT:    adcq %r14, %r12
+; BMI2-NEXT:    adcq %rbx, %rcx
+; BMI2-NEXT:    adcq %r11, %rbp
+; BMI2-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Folded Reload
+; BMI2-NEXT:    adcq %rsi, %r13
+; BMI2-NEXT:    movq %rdx, (%rdi)
+; BMI2-NEXT:    movq %r10, 8(%rdi)
+; BMI2-NEXT:    movq %r9, 16(%rdi)
+; BMI2-NEXT:    movq %r12, 24(%rdi)
+; BMI2-NEXT:    movq %rcx, 32(%rdi)
+; BMI2-NEXT:    movq %rbp, 40(%rdi)
+; BMI2-NEXT:    movq %r8, 48(%rdi)
+; BMI2-NEXT:    movq %r13, 56(%rdi)
+; BMI2-NEXT:    popq %rbx
+; BMI2-NEXT:    .cfi_def_cfa_offset 48
+; BMI2-NEXT:    popq %r12
+; BMI2-NEXT:    .cfi_def_cfa_offset 40
+; BMI2-NEXT:    popq %r13
+; BMI2-NEXT:    .cfi_def_cfa_offset 32
+; BMI2-NEXT:    popq %r14
+; BMI2-NEXT:    .cfi_def_cfa_offset 24
+; BMI2-NEXT:    popq %r15
+; BMI2-NEXT:    .cfi_def_cfa_offset 16
+; BMI2-NEXT:    popq %rbp
+; BMI2-NEXT:    .cfi_def_cfa_offset 8
+; BMI2-NEXT:    retq
+  %x = load i256, ptr %in
+  %z = zext i256 %x to i512
+  %r = mul i512 %z, %z
+  store i512 %r, ptr %out
+  ret void
+}
+
+; full square through a promoted type (i768 -> i1024), 6 limbs -> 12 limbs
+define void @sqr_i768_zext_i384(ptr %out, ptr %in) {
+; X64-LABEL: sqr_i768_zext_i384:
+; X64:       # %bb.0:
+; X64-NEXT:    pushq %rbp
+; X64-NEXT:    .cfi_def_cfa_offset 16
+; X64-NEXT:    pushq %r15
+; X64-NEXT:    .cfi_def_cfa_offset 24
+; X64-NEXT:    pushq %r14
+; X64-NEXT:    .cfi_def_cfa_offset 32
+; X64-NEXT:    pushq %r13
+; X64-NEXT:    .cfi_def_cfa_offset 40
+; X64-NEXT:    pushq %r12
+; X64-NEXT:    .cfi_def_cfa_offset 48
+; X64-NEXT:    pushq %rbx
+; X64-NEXT:    .cfi_def_cfa_offset 56
+; X64-NEXT:    .cfi_offset %rbx, -56
+; X64-NEXT:    .cfi_offset %r12, -48
+; X64-NEXT:    .cfi_offset %r13, -40
+; X64-NEXT:    .cfi_offset %r14, -32
+; X64-NEXT:    .cfi_offset %r15, -24
+; X64-NEXT:    .cfi_offset %rbp, -16
+; X64-NEXT:    movq %rdi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq 24(%rsi), %r12
+; X64-NEXT:    movq 16(%rsi), %r13
+; X64-NEXT:    movq %r13, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq 32(%rsi), %r15
+; X64-NEXT:    movq (%rsi), %r14
+; X64-NEXT:    movq 8(%rsi), %r9
+; X64-NEXT:    movq 40(%rsi), %rdi
+; X64-NEXT:    movq %r9, %rax
+; X64-NEXT:    mulq %rdi
+; X64-NEXT:    movq %rdx, %r10
+; X64-NEXT:    movq %rax, %rcx
+; X64-NEXT:    movq %r14, %rax
+; X64-NEXT:    mulq %r15
+; X64-NEXT:    movq %rdx, %rsi
+; X64-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq %r14, %rax
+; X64-NEXT:    movq %r14, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    mulq %rdi
+; X64-NEXT:    movq %rdx, %r11
+; X64-NEXT:    movq %rax, %rbx
+; X64-NEXT:    addq %rsi, %rbx
+; X64-NEXT:    adcq %rcx, %r11
+; X64-NEXT:    adcq $0, %r10
+; X64-NEXT:    movq %r13, %rax
+; X64-NEXT:    mulq %rdi
+; X64-NEXT:    movq %rdi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq %rdx, %r13
+; X64-NEXT:    movq %rax, %rbp
+; X64-NEXT:    movq %r9, %rax
+; X64-NEXT:    movq %r9, %rcx
+; X64-NEXT:    movq %r9, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    mulq %r15
+; X64-NEXT:    movq %r15, %rsi
+; X64-NEXT:    movq %r15, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq %rdx, %r8
+; X64-NEXT:    movq %rax, %r9
+; X64-NEXT:    movq %r14, %rax
+; X64-NEXT:    mulq %r12
+; X64-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq %rdx, %r14
+; X64-NEXT:    addq {{[-0-9]+}}(%r{{[sb]}}p), %r14 # 8-byte Folded Reload
+; X64-NEXT:    adcq %rbx, %r9
+; X64-NEXT:    adcq %r11, %r8
+; X64-NEXT:    adcq %r10, %rbp
+; X64-NEXT:    movq %rbp, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    adcq $0, %r13
+; X64-NEXT:    movq %r13, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq %r12, %rax
+; X64-NEXT:    movq %r12, %r13
+; X64-NEXT:    mulq %rdi
+; X64-NEXT:    movq %rdx, %r12
+; X64-NEXT:    movq %rax, %r15
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rbp # 8-byte Reload
+; X64-NEXT:    movq %rbp, %rax
+; X64-NEXT:    mulq %rsi
+; X64-NEXT:    movq %rdx, %r11
+; X64-NEXT:    movq %rax, %rsi
+; X64-NEXT:    movq %rcx, %rax
+; X64-NEXT:    mulq %r13
+; X64-NEXT:    movq %rdx, %rcx
+; X64-NEXT:    movq %rax, %r10
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdi # 8-byte Reload
+; X64-NEXT:    movq %rdi, %rax
+; X64-NEXT:    mulq %rbp
+; X64-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq %rdx, %rbx
+; X64-NEXT:    addq {{[-0-9]+}}(%r{{[sb]}}p), %rbx # 8-byte Folded Reload
+; X64-NEXT:    adcq %r14, %r10
+; X64-NEXT:    adcq %r9, %rcx
+; X64-NEXT:    movq %rcx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    adcq %r8, %rsi
+; X64-NEXT:    movq %rsi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %r11 # 8-byte Folded Reload
+; X64-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %r15 # 8-byte Folded Reload
+; X64-NEXT:    movq %r15, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    adcq $0, %r12
+; X64-NEXT:    movq %r12, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r14 # 8-byte Reload
+; X64-NEXT:    movq %r14, %rax
+; X64-NEXT:    mulq {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Folded Reload
+; X64-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq %r13, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq %r13, %rax
+; X64-NEXT:    mulq %r14
+; X64-NEXT:    movq %rdx, %r15
+; X64-NEXT:    movq %rax, %r12
+; X64-NEXT:    movq %rbp, %rsi
+; X64-NEXT:    movq %rbp, %rax
+; X64-NEXT:    mulq %r13
+; X64-NEXT:    movq %rdx, %rbp
+; X64-NEXT:    movq %rax, %rcx
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Reload
+; X64-NEXT:    movq %r9, %rax
+; X64-NEXT:    mulq %rsi
+; X64-NEXT:    movq %rdx, %rsi
+; X64-NEXT:    movq %rax, %r8
+; X64-NEXT:    movq %rdi, %rax
+; X64-NEXT:    mulq %r9
+; X64-NEXT:    movq %rax, %r13
+; X64-NEXT:    movq %rdx, %r9
+; X64-NEXT:    addq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Folded Reload
+; X64-NEXT:    adcq %rbx, %r8
+; X64-NEXT:    adcq %r10, %rsi
+; X64-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %rcx # 8-byte Folded Reload
+; X64-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %rbp # 8-byte Folded Reload
+; X64-NEXT:    adcq %r11, %r12
+; X64-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %r15 # 8-byte Folded Reload
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; X64-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Folded Reload
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; X64-NEXT:    adcq $0, %rax
+; X64-NEXT:    movq %rax, %rdi
+; X64-NEXT:    shldq $1, %rdx, %rdi
+; X64-NEXT:    movq %rdi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    shldq $1, %r15, %rdx
+; X64-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    shldq $1, %r12, %r15
+; X64-NEXT:    shldq $1, %rbp, %r12
+; X64-NEXT:    shrq $63, %rax
+; X64-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    shldq $1, %rcx, %rbp
+; X64-NEXT:    shldq $1, %rsi, %rcx
+; X64-NEXT:    shldq $1, %r8, %rsi
+; X64-NEXT:    shldq $1, %r9, %r8
+; X64-NEXT:    shldq $1, %r13, %r9
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; X64-NEXT:    mulq %rax
+; X64-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq %r14, %rax
+; X64-NEXT:    mulq %r14
+; X64-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; X64-NEXT:    mulq %rax
+; X64-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; X64-NEXT:    movq %rax, %rbx
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; X64-NEXT:    mulq %rax
+; X64-NEXT:    movq %rdx, %r11
+; X64-NEXT:    movq %rax, %r14
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; X64-NEXT:    mulq %rax
+; X64-NEXT:    movq %rdx, %r10
+; X64-NEXT:    movq %rax, %rdi
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; X64-NEXT:    mulq %rax
+; X64-NEXT:    addq %r13, %r13
+; X64-NEXT:    addq %rdx, %r13
+; X64-NEXT:    adcq %r9, %rdi
+; X64-NEXT:    adcq %r8, %r10
+; X64-NEXT:    adcq %rsi, %r14
+; X64-NEXT:    adcq %rcx, %r11
+; X64-NEXT:    adcq %rbp, %rbx
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rbp # 8-byte Reload
+; X64-NEXT:    adcq %r12, %rbp
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Reload
+; X64-NEXT:    adcq %r15, %r9
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Reload
+; X64-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Folded Reload
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rsi # 8-byte Reload
+; X64-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %rsi # 8-byte Folded Reload
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; X64-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Folded Reload
+; X64-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rcx # 8-byte Reload
+; X64-NEXT:    movq %rax, (%rcx)
+; X64-NEXT:    movq %r13, 8(%rcx)
+; X64-NEXT:    movq %rdi, 16(%rcx)
+; X64-NEXT:    movq %r10, 24(%rcx)
+; X64-NEXT:    movq %r14, 32(%rcx)
+; X64-NEXT:    movq %r11, 40(%rcx)
+; X64-NEXT:    movq %rbx, 48(%rcx)
+; X64-NEXT:    movq %rbp, 56(%rcx)
+; X64-NEXT:    movq %r9, 64(%rcx)
+; X64-NEXT:    movq %r8, 72(%rcx)
+; X64-NEXT:    movq %rsi, 80(%rcx)
+; X64-NEXT:    movq %rdx, 88(%rcx)
+; X64-NEXT:    popq %rbx
+; X64-NEXT:    .cfi_def_cfa_offset 48
+; X64-NEXT:    popq %r12
+; X64-NEXT:    .cfi_def_cfa_offset 40
+; X64-NEXT:    popq %r13
+; X64-NEXT:    .cfi_def_cfa_offset 32
+; X64-NEXT:    popq %r14
+; X64-NEXT:    .cfi_def_cfa_offset 24
+; X64-NEXT:    popq %r15
+; X64-NEXT:    .cfi_def_cfa_offset 16
+; X64-NEXT:    popq %rbp
+; X64-NEXT:    .cfi_def_cfa_offset 8
+; X64-NEXT:    retq
+;
+; BMI2-LABEL: sqr_i768_zext_i384:
+; BMI2:       # %bb.0:
+; BMI2-NEXT:    pushq %rbp
+; BMI2-NEXT:    .cfi_def_cfa_offset 16
+; BMI2-NEXT:    pushq %r15
+; BMI2-NEXT:    .cfi_def_cfa_offset 24
+; BMI2-NEXT:    pushq %r14
+; BMI2-NEXT:    .cfi_def_cfa_offset 32
+; BMI2-NEXT:    pushq %r13
+; BMI2-NEXT:    .cfi_def_cfa_offset 40
+; BMI2-NEXT:    pushq %r12
+; BMI2-NEXT:    .cfi_def_cfa_offset 48
+; BMI2-NEXT:    pushq %rbx
+; BMI2-NEXT:    .cfi_def_cfa_offset 56
+; BMI2-NEXT:    .cfi_offset %rbx, -56
+; BMI2-NEXT:    .cfi_offset %r12, -48
+; BMI2-NEXT:    .cfi_offset %r13, -40
+; BMI2-NEXT:    .cfi_offset %r14, -32
+; BMI2-NEXT:    .cfi_offset %r15, -24
+; BMI2-NEXT:    .cfi_offset %rbp, -16
+; BMI2-NEXT:    movq %rdi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq 24(%rsi), %r10
+; BMI2-NEXT:    movq %r10, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq 16(%rsi), %r13
+; BMI2-NEXT:    movq 32(%rsi), %rbp
+; BMI2-NEXT:    movq (%rsi), %r15
+; BMI2-NEXT:    movq 8(%rsi), %rbx
+; BMI2-NEXT:    movq 40(%rsi), %r8
+; BMI2-NEXT:    movq %rbx, %rdx
+; BMI2-NEXT:    movq %rbx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    mulxq %r8, %rax, %rsi
+; BMI2-NEXT:    movq %r15, %rdx
+; BMI2-NEXT:    mulxq %rbp, %rdx, %rcx
+; BMI2-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq %rbp, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq %r15, %rdx
+; BMI2-NEXT:    mulxq %r8, %r9, %r11
+; BMI2-NEXT:    addq %rcx, %r9
+; BMI2-NEXT:    adcq %rax, %r11
+; BMI2-NEXT:    adcq $0, %rsi
+; BMI2-NEXT:    movq %r13, %rdx
+; BMI2-NEXT:    mulxq %r8, %rcx, %rax
+; BMI2-NEXT:    movq %rbx, %rdx
+; BMI2-NEXT:    mulxq %rbp, %rbx, %r14
+; BMI2-NEXT:    movq %r15, %rdx
+; BMI2-NEXT:    movq %r15, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    mulxq %r10, %rdi, %r12
+; BMI2-NEXT:    addq {{[-0-9]+}}(%r{{[sb]}}p), %r12 # 8-byte Folded Reload
+; BMI2-NEXT:    adcq %r9, %rbx
+; BMI2-NEXT:    adcq %r11, %r14
+; BMI2-NEXT:    adcq %rsi, %rcx
+; BMI2-NEXT:    movq %rcx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    adcq $0, %rax
+; BMI2-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq %r10, %rdx
+; BMI2-NEXT:    movq %r8, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    mulxq %r8, %rdx, %r9
+; BMI2-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq %r13, %rdx
+; BMI2-NEXT:    mulxq %rbp, %rsi, %rcx
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; BMI2-NEXT:    mulxq %r10, %rax, %r11
+; BMI2-NEXT:    movq %r15, %rdx
+; BMI2-NEXT:    mulxq %r13, %rdx, %r10
+; BMI2-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    addq %rdi, %r10
+; BMI2-NEXT:    adcq %r12, %rax
+; BMI2-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    adcq %rbx, %r11
+; BMI2-NEXT:    adcq %r14, %rsi
+; BMI2-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %rcx # 8-byte Folded Reload
+; BMI2-NEXT:    movq %rcx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; BMI2-NEXT:    adcq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Folded Spill
+; BMI2-NEXT:    adcq $0, %r9
+; BMI2-NEXT:    movq %r9, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq %rbp, %rdx
+; BMI2-NEXT:    mulxq %r8, %rbx, %rax
+; BMI2-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Reload
+; BMI2-NEXT:    movq %r9, %rdx
+; BMI2-NEXT:    mulxq %rbp, %r12, %r15
+; BMI2-NEXT:    movq %r13, %rax
+; BMI2-NEXT:    movq %r13, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq %r13, %rdx
+; BMI2-NEXT:    mulxq %r9, %rbp, %r13
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Reload
+; BMI2-NEXT:    movq %r9, %rdx
+; BMI2-NEXT:    mulxq %rax, %rcx, %rdi
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; BMI2-NEXT:    mulxq %r9, %r14, %r8
+; BMI2-NEXT:    addq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Folded Reload
+; BMI2-NEXT:    adcq %r10, %rcx
+; BMI2-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %rdi # 8-byte Folded Reload
+; BMI2-NEXT:    adcq %r11, %rbp
+; BMI2-NEXT:    adcq %rsi, %r13
+; BMI2-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %r12 # 8-byte Folded Reload
+; BMI2-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %r15 # 8-byte Folded Reload
+; BMI2-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %rbx # 8-byte Folded Reload
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; BMI2-NEXT:    adcq $0, %rdx
+; BMI2-NEXT:    movq %rdx, %rsi
+; BMI2-NEXT:    shldq $1, %rbx, %rsi
+; BMI2-NEXT:    movq %rsi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    shldq $1, %r15, %rbx
+; BMI2-NEXT:    shldq $1, %r12, %r15
+; BMI2-NEXT:    shldq $1, %r13, %r12
+; BMI2-NEXT:    shrq $63, %rdx
+; BMI2-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    shldq $1, %rbp, %r13
+; BMI2-NEXT:    shldq $1, %rdi, %rbp
+; BMI2-NEXT:    shldq $1, %rcx, %rdi
+; BMI2-NEXT:    shldq $1, %r8, %rcx
+; BMI2-NEXT:    shldq $1, %r14, %r8
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; BMI2-NEXT:    mulxq %rdx, %rsi, %rdx
+; BMI2-NEXT:    movq %rsi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; BMI2-NEXT:    mulxq %rdx, %rsi, %rdx
+; BMI2-NEXT:    movq %rsi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; BMI2-NEXT:    mulxq %rdx, %rdx, %rax
+; BMI2-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; BMI2-NEXT:    mulxq %rdx, %r11, %rax
+; BMI2-NEXT:    movq %r9, %rdx
+; BMI2-NEXT:    mulxq %r9, %r9, %rsi
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; BMI2-NEXT:    mulxq %rdx, %r10, %rdx
+; BMI2-NEXT:    movq %r10, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; BMI2-NEXT:    addq %r14, %r14
+; BMI2-NEXT:    addq %rdx, %r14
+; BMI2-NEXT:    adcq %r8, %r9
+; BMI2-NEXT:    adcq %rcx, %rsi
+; BMI2-NEXT:    adcq %rdi, %r11
+; BMI2-NEXT:    adcq %rbp, %rax
+; BMI2-NEXT:    movq %rax, %rdi
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rbp # 8-byte Reload
+; BMI2-NEXT:    adcq %r13, %rbp
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r13 # 8-byte Reload
+; BMI2-NEXT:    adcq %r12, %r13
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Reload
+; BMI2-NEXT:    adcq %r15, %r8
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; BMI2-NEXT:    adcq %rbx, %rdx
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rcx # 8-byte Reload
+; BMI2-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %rcx # 8-byte Folded Reload
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; BMI2-NEXT:    adcq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Folded Reload
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r10 # 8-byte Reload
+; BMI2-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rbx # 8-byte Reload
+; BMI2-NEXT:    movq %r10, (%rbx)
+; BMI2-NEXT:    movq %r14, 8(%rbx)
+; BMI2-NEXT:    movq %r9, 16(%rbx)
+; BMI2-NEXT:    movq %rsi, 24(%rbx)
+; BMI2-NEXT:    movq %r11, 32(%rbx)
+; BMI2-NEXT:    movq %rdi, 40(%rbx)
+; BMI2-NEXT:    movq %rbp, 48(%rbx)
+; BMI2-NEXT:    movq %r13, 56(%rbx)
+; BMI2-NEXT:    movq %r8, 64(%rbx)
+; BMI2-NEXT:    movq %rdx, 72(%rbx)
+; BMI2-NEXT:    movq %rcx, 80(%rbx)
+; BMI2-NEXT:    movq %rax, 88(%rbx)
+; BMI2-NEXT:    popq %rbx
+; BMI2-NEXT:    .cfi_def_cfa_offset 48
+; BMI2-NEXT:    popq %r12
+; BMI2-NEXT:    .cfi_def_cfa_offset 40
+; BMI2-NEXT:    popq %r13
+; BMI2-NEXT:    .cfi_def_cfa_offset 32
+; BMI2-NEXT:    popq %r14
+; BMI2-NEXT:    .cfi_def_cfa_offset 24
+; BMI2-NEXT:    popq %r15
+; BMI2-NEXT:    .cfi_def_cfa_offset 16
+; BMI2-NEXT:    popq %rbp
+; BMI2-NEXT:    .cfi_def_cfa_offset 8
+; BMI2-NEXT:    retq
+  %x = load i384, ptr %in
+  %z = zext i384 %x to i768
+  %r = mul i768 %z, %z
+  store i768 %r, ptr %out
+  ret void
+}
+
+; sign-extended operand: no known-zero limbs, truncated 4 x 4
+define void @sqr_i256_sext_i128(ptr %out, ptr %in) {
+; X64-LABEL: sqr_i256_sext_i128:
+; X64:       # %bb.0:
+; X64-NEXT:    pushq %rbx
+; X64-NEXT:    .cfi_def_cfa_offset 16
+; X64-NEXT:    .cfi_offset %rbx, -16
+; X64-NEXT:    movq (%rsi), %r9
+; X64-NEXT:    movq 8(%rsi), %rsi
+; X64-NEXT:    movq %rsi, %rcx
+; X64-NEXT:    sarq $63, %rcx
+; X64-NEXT:    movq %rsi, %rbx
+; X64-NEXT:    imulq %rcx, %rbx
+; X64-NEXT:    movq %r9, %rax
+; X64-NEXT:    mulq %rcx
+; X64-NEXT:    movq %rdx, %rcx
+; X64-NEXT:    movq %rax, %r10
+; X64-NEXT:    addq %rax, %rcx
+; X64-NEXT:    movq %r9, %rax
+; X64-NEXT:    mulq %rsi
+; X64-NEXT:    movq %rax, %r8
+; X64-NEXT:    movq %rdx, %r11
+; X64-NEXT:    addq %r10, %r11
+; X64-NEXT:    adcq %rbx, %rcx
+; X64-NEXT:    shldq $1, %r11, %rcx
+; X64-NEXT:    shldq $1, %rax, %r11
+; X64-NEXT:    movq %rsi, %rax
+; X64-NEXT:    mulq %rsi
+; X64-NEXT:    movq %rdx, %rsi
+; X64-NEXT:    movq %rax, %r10
+; X64-NEXT:    movq %r9, %rax
+; X64-NEXT:    mulq %r9
+; X64-NEXT:    addq %r8, %r8
+; X64-NEXT:    addq %rdx, %r8
+; X64-NEXT:    adcq %r11, %r10
+; X64-NEXT:    adcq %rcx, %rsi
+; X64-NEXT:    movq %rax, (%rdi)
+; X64-NEXT:    movq %r8, 8(%rdi)
+; X64-NEXT:    movq %r10, 16(%rdi)
+; X64-NEXT:    movq %rsi, 24(%rdi)
+; X64-NEXT:    popq %rbx
+; X64-NEXT:    .cfi_def_cfa_offset 8
+; X64-NEXT:    retq
+;
+; BMI2-LABEL: sqr_i256_sext_i128:
+; BMI2:       # %bb.0:
+; BMI2-NEXT:    pushq %r14
+; BMI2-NEXT:    .cfi_def_cfa_offset 16
+; BMI2-NEXT:    pushq %rbx
+; BMI2-NEXT:    .cfi_def_cfa_offset 24
+; BMI2-NEXT:    .cfi_offset %rbx, -24
+; BMI2-NEXT:    .cfi_offset %r14, -16
+; BMI2-NEXT:    movq (%rsi), %rax
+; BMI2-NEXT:    movq 8(%rsi), %rcx
+; BMI2-NEXT:    movq %rcx, %r8
+; BMI2-NEXT:    sarq $63, %r8
+; BMI2-NEXT:    movq %rax, %rdx
+; BMI2-NEXT:    mulxq %r8, %r9, %rsi
+; BMI2-NEXT:    mulxq %rcx, %r11, %r10
+; BMI2-NEXT:    movq %rcx, %rdx
+; BMI2-NEXT:    mulxq %rcx, %rbx, %r14
+; BMI2-NEXT:    imulq %r8, %rcx
+; BMI2-NEXT:    addq %r9, %rsi
+; BMI2-NEXT:    addq %r9, %r10
+; BMI2-NEXT:    adcq %rcx, %rsi
+; BMI2-NEXT:    shldq $1, %r10, %rsi
+; BMI2-NEXT:    shldq $1, %r11, %r10
+; BMI2-NEXT:    movq %rax, %rdx
+; BMI2-NEXT:    mulxq %rax, %rcx, %rax
+; BMI2-NEXT:    addq %r11, %r11
+; BMI2-NEXT:    addq %rax, %r11
+; BMI2-NEXT:    adcq %r10, %rbx
+; BMI2-NEXT:    adcq %rsi, %r14
+; BMI2-NEXT:    movq %rcx, (%rdi)
+; BMI2-NEXT:    movq %r11, 8(%rdi)
+; BMI2-NEXT:    movq %rbx, 16(%rdi)
+; BMI2-NEXT:    movq %r14, 24(%rdi)
+; BMI2-NEXT:    popq %rbx
+; BMI2-NEXT:    .cfi_def_cfa_offset 16
+; BMI2-NEXT:    popq %r14
+; BMI2-NEXT:    .cfi_def_cfa_offset 8
+; BMI2-NEXT:    retq
+  %x = load i128, ptr %in
+  %z = sext i128 %x to i256
+  %r = mul i256 %z, %z
+  store i256 %r, ptr %out
+  ret void
+}
+
+; a single limb: one product, not expanded here
+define void @sqr_i256_zext_i64(ptr %out, ptr %in) {
+; X64-LABEL: sqr_i256_zext_i64:
+; X64:       # %bb.0:
+; X64-NEXT:    movq (%rsi), %rax
+; X64-NEXT:    mulq %rax
+; X64-NEXT:    xorps %xmm0, %xmm0
+; X64-NEXT:    movaps %xmm0, 16(%rdi)
+; X64-NEXT:    movq %rdx, 8(%rdi)
+; X64-NEXT:    movq %rax, (%rdi)
+; X64-NEXT:    retq
+;
+; BMI2-LABEL: sqr_i256_zext_i64:
+; BMI2:       # %bb.0:
+; BMI2-NEXT:    movq (%rsi), %rdx
+; BMI2-NEXT:    mulxq %rdx, %rcx, %rax
+; BMI2-NEXT:    xorps %xmm0, %xmm0
+; BMI2-NEXT:    movaps %xmm0, 16(%rdi)
+; BMI2-NEXT:    movq %rax, 8(%rdi)
+; BMI2-NEXT:    movq %rcx, (%rdi)
+; BMI2-NEXT:    retq
+  %x = load i64, ptr %in
+  %z = zext i64 %x to i256
+  %r = mul i256 %z, %z
+  store i256 %r, ptr %out
+  ret void
+}