[llvm] [InstCombine] visitShuffleVectorInst assert with vector of pointers fix. (PR #152341)

[Szymon Piotr Milczek via llvm-commits]

https://github.com/smilczek updated https://github.com/llvm/llvm-project/pull/152341

>From 9d3e8bd830edf6454e25f6f03801e55593c2aa5d Mon Sep 17 00:00:00 2001
From: "Milczek, Szymon" <szymon.milczek at intel.com>
Date: Wed, 6 Aug 2025 18:28:23 +0200
Subject: [PATCH 1/3] [InstCombine] visitShuffleVectorInst assert with vector
 of pointers fix.

In visitShuffleVectorInst there's an if block that's meant to turn
shufflevector followed by bitcast into extractelement where possible.

It assumes that there will never be bitcasts performed on vectors of ptr
as such operations are almost always illegal, and ptrtoint instructions
should be used instead.

There is however an edge case where a bitcast instruction can be
performed on a vector of type `<1 x ptr>` to turn it into type `ptr`

In this edge case, the code initializes the variable `VecBitWidth` to 0.
Then, when iterating over users that are bitcasts, an attempt is made to
create a vector of size 0, which triggers and assert.

This commit adds a condition to the if statement, that ensures the logic
doesn't enter the if block for a vector of ptr. This approach assumes,
that there won't be any work to do on a vector of ptr, as there's only
one valid case where a bitcast is performed on such a vector and that
edge case is already handled in visitBitCast method.
---
 llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp | 3 ++-
 ...025-08-06-shufflevector-bitcast-vector-of-pointers.ll | 9 +++++++++
 2 files changed, 11 insertions(+), 1 deletion(-)
 create mode 100644 llvm/test/Transforms/InstCombine/2025-08-06-shufflevector-bitcast-vector-of-pointers.ll

diff --git a/llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp b/llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp
index fe0f308223387..510c2cf0bed04 100644
--- a/llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp
+++ b/llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp
@@ -3038,7 +3038,8 @@ Instruction *InstCombinerImpl::visitShuffleVectorInst(ShuffleVectorInst &SVI) {
   // Index range [6,10):              ^-----------^ Needs an extra shuffle.
   // Target type i40:           ^--------------^ Won't work, bail.
   bool MadeChange = false;
-  if (isShuffleExtractingFromLHS(SVI, Mask)) {
+  if (isShuffleExtractingFromLHS(SVI, Mask) &&
+      !LHS->getType()->isPtrOrPtrVectorTy()) {
     Value *V = LHS;
     unsigned MaskElems = Mask.size();
     auto *SrcTy = cast<FixedVectorType>(V->getType());
diff --git a/llvm/test/Transforms/InstCombine/2025-08-06-shufflevector-bitcast-vector-of-pointers.ll b/llvm/test/Transforms/InstCombine/2025-08-06-shufflevector-bitcast-vector-of-pointers.ll
new file mode 100644
index 0000000000000..a035dfbfceaa0
--- /dev/null
+++ b/llvm/test/Transforms/InstCombine/2025-08-06-shufflevector-bitcast-vector-of-pointers.ll
@@ -0,0 +1,9 @@
+; RUN: opt < %s -passes=instcombine -S
+
+; Make sure that we don't crash when optimizing shufflevector of <N x ptr> with <1 x i32> mask followed by bitcast of <1 x ptr> to ptr
+
+define ptr @test(<3 x ptr> %vptr) {
+  %SV = shufflevector <3 x ptr> %vptr, <3 x ptr> zeroinitializer, <1 x i32> zeroinitializer
+  %BC = bitcast <1 x ptr> %SV to ptr
+  ret ptr %BC
+}

>From 6786ea40f92b1e83b3b6e6b239d77147f81e4c82 Mon Sep 17 00:00:00 2001
From: "Milczek, Szymon" <szymon.milczek at intel.com>
Date: Thu, 7 Aug 2025 11:57:20 +0200
Subject: [PATCH 2/3] don't bail on vector of ptr

---
 llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp b/llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp
index 510c2cf0bed04..b17cf17db1580 100644
--- a/llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp
+++ b/llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp
@@ -3038,12 +3038,11 @@ Instruction *InstCombinerImpl::visitShuffleVectorInst(ShuffleVectorInst &SVI) {
   // Index range [6,10):              ^-----------^ Needs an extra shuffle.
   // Target type i40:           ^--------------^ Won't work, bail.
   bool MadeChange = false;
-  if (isShuffleExtractingFromLHS(SVI, Mask) &&
-      !LHS->getType()->isPtrOrPtrVectorTy()) {
+  if (isShuffleExtractingFromLHS(SVI, Mask)) {
     Value *V = LHS;
     unsigned MaskElems = Mask.size();
     auto *SrcTy = cast<FixedVectorType>(V->getType());
-    unsigned VecBitWidth = SrcTy->getPrimitiveSizeInBits().getFixedValue();
+    unsigned VecBitWidth = DL.getTypeSizeInBits(SrcTy);
     unsigned SrcElemBitWidth = DL.getTypeSizeInBits(SrcTy->getElementType());
     assert(SrcElemBitWidth && "vector elements must have a bitwidth");
     unsigned SrcNumElems = SrcTy->getNumElements();

>From 2c0ea1af7efa3f7e7883f733fba81d94e8199bc2 Mon Sep 17 00:00:00 2001
From: "Milczek, Szymon" <szymon.milczek at intel.com>
Date: Thu, 7 Aug 2025 11:58:12 +0200
Subject: [PATCH 3/3] run update_test_checks.py

---
 .../2025-08-06-shufflevector-bitcast-vector-of-pointers.ll       | 1 +
 1 file changed, 1 insertion(+)

diff --git a/llvm/test/Transforms/InstCombine/2025-08-06-shufflevector-bitcast-vector-of-pointers.ll b/llvm/test/Transforms/InstCombine/2025-08-06-shufflevector-bitcast-vector-of-pointers.ll
index a035dfbfceaa0..babb2fa2fae7d 100644
--- a/llvm/test/Transforms/InstCombine/2025-08-06-shufflevector-bitcast-vector-of-pointers.ll
+++ b/llvm/test/Transforms/InstCombine/2025-08-06-shufflevector-bitcast-vector-of-pointers.ll
@@ -1,3 +1,4 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 5
 ; RUN: opt < %s -passes=instcombine -S
 
 ; Make sure that we don't crash when optimizing shufflevector of <N x ptr> with <1 x i32> mask followed by bitcast of <1 x ptr> to ptr