[all-commits] [llvm/llvm-project] 7c3cf4: [Inline][X86] Avoid inlining if it would create AB...

[Nikita Popov via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 7c3cf4c2c0689be1a08b8a1326703ec5770de471
      https://github.com/llvm/llvm-project/commit/7c3cf4c2c0689be1a08b8a1326703ec5770de471
  Author: Nikita Popov <npopov at redhat.com>
  Date:   2021-12-27 (Mon, 27 Dec 2021)

  Changed paths:
    M llvm/lib/Target/X86/X86TargetTransformInfo.cpp
    M llvm/test/Transforms/Inline/X86/call-abi-compatibility.ll

  Log Message:
  -----------
  [Inline][X86] Avoid inlining if it would create ABI-incompatible calls (PR52660)

X86 allows inlining functions if the callee target features are a
subset of the caller target features. This ensures that we don't
inline something into a caller that does not support it.

However, this does not account for possible call ABI mismatches as
a result of inlining. If a call passing a vector argument was
originally in a -avx function, calling another -avx function, the
vector is passed in xmm. If we now inline it into a +avx function,
then it will be passed in ymm, even though the callee expects it in xmm.

Fix this by scanning over all calls in the function and checking
whether ABI incompatibility is possible. Calls that only pass scalar
types are excluded, as I believe those always use the same ABI
independent of target features.

Fixes https://github.com/llvm/llvm-project/issues/52660.

Differential Revision: https://reviews.llvm.org/D116036