[llvm] Reapply "[SelectionDAG] Recurse through mask expression trees in WidenVSELECTMask" (PR #217307)

[Mikael Holmen via llvm-commits]

mikaelholmen wrote:

Hello @SavchenkoValeriy 
The following starts crashing with this patch:
```
llc -mtriple=x86_64-unknown-linux-gnu bbi-121093.ll
```
It crashes like
```
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
Stack dump:
0.	Program arguments: build-all/bin/llc -mtriple=x86_64-unknown-linux-gnu bbi-121093.ll
1.	Running pass 'Function Pass Manager' on module 'bbi-121093.ll'.
2.	Running pass 'X86 DAG->DAG Instruction Selection' on function '@f'
 #0 0x0000558efed621bc llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (build-all/bin/llc+0x57111bc)
 #1 0x0000558efed5ee61 llvm::sys::RunSignalHandlers() (build-all/bin/llc+0x570de61)
 #2 0x0000558efed62c83 SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
 #3 0x00007fa46ba0f990 __restore_rt (/lib64/libpthread.so.0+0x12990)
 #4 0x0000558eff245406 llvm::FixedVectorType::get(llvm::Type*, unsigned int) (build-all/bin/llc+0x5bf4406)
 #5 0x0000558effa2497b llvm::EVT::getExtendedVectorVT(llvm::LLVMContext&, llvm::EVT, llvm::ElementCount) (build-all/bin/llc+0x63d397b)
 #6 0x0000558f0023a283 llvm::EVT::changeVectorElementType(llvm::LLVMContext&, llvm::EVT) const SelectionDAGBuilder.cpp:0:0
 #7 0x0000558f004d745b llvm::DAGTypeLegalizer::unifyMaskTypes(llvm::SDValue&, bool, llvm::SDValue&, bool, llvm::EVT) LegalizeVectorTypes.cpp:0:0
 #8 0x0000558f004d77d8 llvm::DAGTypeLegalizer::convertMaskTreeImpl(llvm::SDValue, llvm::EVT, unsigned int) LegalizeVectorTypes.cpp:0:0
 #9 0x0000558f004d7df0 llvm::DAGTypeLegalizer::convertMaskTree(llvm::SDValue, llvm::EVT) LegalizeVectorTypes.cpp:0:0
#10 0x0000558f004d81d6 llvm::DAGTypeLegalizer::WidenVSELECTMask(llvm::SDNode*) LegalizeVectorTypes.cpp:0:0
#11 0x0000558f004c58d8 llvm::DAGTypeLegalizer::WidenVecRes_Select(llvm::SDNode*) LegalizeVectorTypes.cpp:0:0
#12 0x0000558f004c0973 llvm::DAGTypeLegalizer::WidenVectorResult(llvm::SDNode*, unsigned int) LegalizeVectorTypes.cpp:0:0
#13 0x0000558f00456b1c llvm::DAGTypeLegalizer::run() LegalizeTypes.cpp:0:0
#14 0x0000558f0045b903 llvm::SelectionDAG::LegalizeTypes() (build-all/bin/llc+0x6e0a903)
#15 0x0000558f0040425e llvm::SelectionDAGISel::CodeGenAndEmitDAG() (build-all/bin/llc+0x6db325e)
#16 0x0000558f00402c3c llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&) (build-all/bin/llc+0x6db1c3c)
#17 0x0000558f004001c2 llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&) (build-all/bin/llc+0x6daf1c2)
#18 0x0000558f003fd869 llvm::SelectionDAGISelLegacy::runOnMachineFunction(llvm::MachineFunction&) (build-all/bin/llc+0x6dac869)
#19 0x0000558eff780fae llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (build-all/bin/llc+0x612ffae)
#20 0x0000558eff421437 llvm::FPPassManager::runOnFunction(llvm::Function&) (build-all/bin/llc+0x5dd0437)
#21 0x0000558eff429e22 llvm::FPPassManager::runOnModule(llvm::Module&) (build-all/bin/llc+0x5dd8e22)
#22 0x0000558eff42226c llvm::legacy::PassManagerImpl::run(llvm::Module&) (build-all/bin/llc+0x5dd126c)
#23 0x0000558efed69d9d compileModule(char**, llvm::SmallVectorImpl<llvm::PassPlugin>&, llvm::LLVMContext&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&) llcdriver.cpp:0:0
#24 0x0000558efed67111 llcMain (build-all/bin/llc+0x5716111)
#25 0x0000558efec9f2bd main (build-all/bin/llc+0x564e2bd)
#26 0x00007fa469580865 __libc_start_main (/lib64/libc.so.6+0x3a865)
#27 0x0000558efec9f1ae _start (build-all/bin/llc+0x564e1ae)
Segmentation fault (core dumped)
```
[bbi-121093.ll.gz](https://github.com/user-attachments/files/32520992/bbi-121093.ll.gz)


https://github.com/llvm/llvm-project/pull/217307