[PATCH] D128732: [VectorCombine] Improve shuffle select shuffle-of-shuffles

[Bevin Hansson via Phabricator via llvm-commits]

ebevhan added a comment.

Hi!

We get a memory corruption issue in our downstream testing with this patch. I've attached a reproducer, which can be run with `opt -passes=vector-combine reduced.ll`:

F23804137: reduced.ll <https://reviews.llvm.org/F23804137>

The issue occurs in X86TTIImpl::getShuffleCost:

  unsigned E = *NumOfDests.getValue();
  unsigned NormalizedVF =
      LegalVT.getVectorNumElements() * std::max(NumOfSrcs, E);
  unsigned NumOfSrcRegs = NormalizedVF / LegalVT.getVectorNumElements();
  unsigned NumOfDestRegs = NormalizedVF / LegalVT.getVectorNumElements();
  SmallVector<int> NormalizedMask(NormalizedVF, UndefMaskElem);
  copy(Mask, NormalizedMask.begin());

Mask has a size of 64, but NormalizedVF is only 16, so the copy tramples over the heap.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D128732/new/

https://reviews.llvm.org/D128732