[llvm] [SelectionDAG] Do not crash on large integers in CheckInteger (PR #75787)

[Ulrich Weigand via llvm-commits]

https://github.com/uweigand created https://github.com/llvm/llvm-project/pull/75787

The CheckInteger routine called from TableGen-generated selection logic uses getSExtValue - which will abort if the underlying APInt does not fit into an int64_t.

This case is now triggered by the SystemZ back-end since i128 is a legal type on certain machines.  While we do not have any regular instructions that take 128-bit immediates (like most other platforms), there are patterns in the .td files that recognize an i128 "xor ..., -1" as a "not".

These patterns cause code to be generated that calls the CheckInteger routine on some i128-valued integer, which may trigger the assert.

Fix by using trySExtValue instead.

Fixes https://github.com/llvm/llvm-project/issues/75710

>From 85573ffd6e22942c788b749c80f111096424b8ad Mon Sep 17 00:00:00 2001
From: Ulrich Weigand <ulrich.weigand at de.ibm.com>
Date: Mon, 18 Dec 2023 13:00:16 +0100
Subject: [PATCH] [SelectionDAG] Do not crash on large integers in CheckInteger

The CheckInteger routine called from TableGen-generated
selection logic uses getSExtValue - which will abort if
the underlying APInt does not fit into an int64_t.

This case is now triggered by the SystemZ back-end since
i128 is a legal type on certain machines.  While we do not
have any regular instructions that take 128-bit immediates
(like most other platforms), there are patterns in the .td
files that recognize an i128 "xor ..., -1" as a "not".

These patterns cause code to be generated that calls the
CheckInteger routine on some i128-valued integer, which
may trigger the assert.

Fix by using trySExtValue instead.

Fixes https://github.com/llvm/llvm-project/issues/75710
---
 llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp |  2 +-
 llvm/test/CodeGen/SystemZ/xor-09.ll                | 14 ++++++++++++++
 2 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp
index a1cf4cbbee1b85..af49ef17a3f2dd 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp
@@ -2786,7 +2786,7 @@ CheckInteger(const unsigned char *MatcherTable, unsigned &MatcherIndex,
   Val = decodeSignRotatedValue(Val);
 
   ConstantSDNode *C = dyn_cast<ConstantSDNode>(N);
-  return C && C->getSExtValue() == Val;
+  return C && C->getAPIntValue().trySExtValue() == Val;
 }
 
 LLVM_ATTRIBUTE_ALWAYS_INLINE static bool
diff --git a/llvm/test/CodeGen/SystemZ/xor-09.ll b/llvm/test/CodeGen/SystemZ/xor-09.ll
index d0287f7fdd77ec..7b7aaa404c00f6 100644
--- a/llvm/test/CodeGen/SystemZ/xor-09.ll
+++ b/llvm/test/CodeGen/SystemZ/xor-09.ll
@@ -15,3 +15,17 @@ define i128 @f1(i128 %a, i128 %b) {
   %res = xor i128 %a, %b
   ret i128 %res
 }
+
+; Verify that xor with a large constant does not crash.
+define i128 @f2(i128 %x) {
+; CHECK-LABEL: f2:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    larl %r1, .LCPI1_0
+; CHECK-NEXT:    vl %v0, 0(%r3), 3
+; CHECK-NEXT:    vl %v1, 0(%r1), 3
+; CHECK-NEXT:    vx %v0, %v0, %v1
+; CHECK-NEXT:    vst %v0, 0(%r2), 3
+; CHECK-NEXT:    br %r14
+  %res = xor i128 %x, 17440380254424117642
+  ret i128 %res
+}