[PATCH] D116036: [Inline][X86] Avoid inlining if it would create ABI-incompatible calls (PR52660)

[Nikita Popov via Phabricator via llvm-commits]

nikic created this revision.
nikic added reviewers: craig.topper, RKSimon, pengfei, tstellar.
Herald added a subscriber: hiraditya.
nikic requested review of this revision.
Herald added a project: LLVM.
Herald added a subscriber: llvm-commits.

X86 allows inlining functions if the callee target features are a subset of the caller target features. This ensures that we don't inline something into a caller that does not support it.

However, this does not account for possible call ABI mismatches as a result of inlining. If a call passing a vector argument was originally in `-avx` function, calling another `-avx` function, the vector is passed in xmm. If we now inline it into an `+avx` function, then it will be passed in ymm, even though the callee expects it in xmm.

Fix this by scanning over all calls in the function and checking whether ABI incompatibility is possible. Calls that only pass scalar types are excluded, as I believe those always use the same ABI independent of target features (right?)

Fixes https://github.com/llvm/llvm-project/issues/52660.


https://reviews.llvm.org/D116036

Files:
  llvm/lib/Target/X86/X86TargetTransformInfo.cpp
  llvm/test/Transforms/Inline/X86/call-abi-compatibility.ll

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D116036.395423.patch
Type: text/x-patch
Size: 4138 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20211220/674b3878/attachment.bin>