[PATCH] D41719: [InlineCost] Prevent infinite recursion on function pointers

Jessica Paquette via Phabricator via llvm-commits llvm-commits at lists.llvm.org
Wed Jan 3 14:40:44 PST 2018

paquette created this revision.
paquette added reviewers: echristo, davide, MatzeB.
Herald added subscribers: haicheng, eraman.

Compile the following at -Os:

  typedef void (*Foo)(void*);
  void Bar(void* FunctionPtr)
  int main(int argc, char *argv[]) {

The inliner will recurse infinitely because it doesn't handle the case where a function takes a function pointer argument, and is called using a pointer to itself as an argument. This patch makes the inliner quit when the examined callsite involves a function pointer parameter.



Index: lib/Analysis/InlineCost.cpp
--- lib/Analysis/InlineCost.cpp
+++ lib/Analysis/InlineCost.cpp
@@ -1254,6 +1254,17 @@
     return Base::visitCallSite(CS);
+  // Check if the callee takes a function pointer argument. If it does, then the
+  // caller could be passed to the callee, so we should back out.
+  for (Use &U : CS.getInstruction()->operands()) {
+    if (CastInst *CI = dyn_cast<CastInst>(U)) {
+      Type *DestTy = CI->getDestTy();
+      if (DestTy->isPointerTy() &&
+          DestTy->getPointerElementType()->isFunctionTy()) 
+        return false;
+    }
+  }
   // Otherwise we're in a very special case -- an indirect function call. See
   // if we can be particularly clever about this.
   Value *Callee = CS.getCalledValue();

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D41719.128564.patch
Type: text/x-patch
Size: 827 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20180103/dcdaca4f/attachment.bin>

More information about the llvm-commits mailing list