[llvm] [InstCombine] visitShuffleVectorInst assert with vector of pointers fix. (PR #152341)

[Szymon Piotr Milczek via llvm-commits]

https://github.com/smilczek created https://github.com/llvm/llvm-project/pull/152341

In visitShuffleVectorInst there's an if block that's meant to turn
shufflevector followed by bitcast into extractelement where possible.

It assumes that there will never be bitcasts performed on vectors of ptr
as such operations are almost always illegal, and ptrtoint instructions
should be used instead.

There is however an edge case where a bitcast instruction can be
performed on a vector of type `<1 x ptr>` to turn it into type `ptr`

In this edge case, the code initializes the variable `VecBitWidth` to 0.
Then, when iterating over users that are bitcasts, an attempt is made to
create a vector of size 0, which triggers and assert.

This commit adds a condition to the if statement, that ensures the logic
doesn't enter the if block for a vector of ptr. This approach assumes,
that there won't be any work to do on a vector of ptr, as there's only
one valid case where a bitcast is performed on such a vector and that
edge case is already handled in visitBitCast method.

>From 9d3e8bd830edf6454e25f6f03801e55593c2aa5d Mon Sep 17 00:00:00 2001
From: "Milczek, Szymon" <szymon.milczek at intel.com>
Date: Wed, 6 Aug 2025 18:28:23 +0200
Subject: [PATCH] [InstCombine] visitShuffleVectorInst assert with vector of
 pointers fix.

In visitShuffleVectorInst there's an if block that's meant to turn
shufflevector followed by bitcast into extractelement where possible.

It assumes that there will never be bitcasts performed on vectors of ptr
as such operations are almost always illegal, and ptrtoint instructions
should be used instead.

There is however an edge case where a bitcast instruction can be
performed on a vector of type `<1 x ptr>` to turn it into type `ptr`

In this edge case, the code initializes the variable `VecBitWidth` to 0.
Then, when iterating over users that are bitcasts, an attempt is made to
create a vector of size 0, which triggers and assert.

This commit adds a condition to the if statement, that ensures the logic
doesn't enter the if block for a vector of ptr. This approach assumes,
that there won't be any work to do on a vector of ptr, as there's only
one valid case where a bitcast is performed on such a vector and that
edge case is already handled in visitBitCast method.
---
 llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp | 3 ++-
 ...025-08-06-shufflevector-bitcast-vector-of-pointers.ll | 9 +++++++++
 2 files changed, 11 insertions(+), 1 deletion(-)
 create mode 100644 llvm/test/Transforms/InstCombine/2025-08-06-shufflevector-bitcast-vector-of-pointers.ll

diff --git a/llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp b/llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp
index fe0f308223387..510c2cf0bed04 100644
--- a/llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp
+++ b/llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp
@@ -3038,7 +3038,8 @@ Instruction *InstCombinerImpl::visitShuffleVectorInst(ShuffleVectorInst &SVI) {
   // Index range [6,10):              ^-----------^ Needs an extra shuffle.
   // Target type i40:           ^--------------^ Won't work, bail.
   bool MadeChange = false;
-  if (isShuffleExtractingFromLHS(SVI, Mask)) {
+  if (isShuffleExtractingFromLHS(SVI, Mask) &&
+      !LHS->getType()->isPtrOrPtrVectorTy()) {
     Value *V = LHS;
     unsigned MaskElems = Mask.size();
     auto *SrcTy = cast<FixedVectorType>(V->getType());
diff --git a/llvm/test/Transforms/InstCombine/2025-08-06-shufflevector-bitcast-vector-of-pointers.ll b/llvm/test/Transforms/InstCombine/2025-08-06-shufflevector-bitcast-vector-of-pointers.ll
new file mode 100644
index 0000000000000..a035dfbfceaa0
--- /dev/null
+++ b/llvm/test/Transforms/InstCombine/2025-08-06-shufflevector-bitcast-vector-of-pointers.ll
@@ -0,0 +1,9 @@
+; RUN: opt < %s -passes=instcombine -S
+
+; Make sure that we don't crash when optimizing shufflevector of <N x ptr> with <1 x i32> mask followed by bitcast of <1 x ptr> to ptr
+
+define ptr @test(<3 x ptr> %vptr) {
+  %SV = shufflevector <3 x ptr> %vptr, <3 x ptr> zeroinitializer, <1 x i32> zeroinitializer
+  %BC = bitcast <1 x ptr> %SV to ptr
+  ret ptr %BC
+}