[PATCH] D41719: [InlineCost] Prevent infinite recursion on function pointers

[Jessica Paquette via Phabricator via llvm-commits]

paquette created this revision.
paquette added reviewers: echristo, davide, MatzeB.
Herald added subscribers: haicheng, eraman.

Compile the following at -Os:

  typedef void (*Foo)(void*);
  
  void Bar(void* FunctionPtr)
  {
  	((Foo)FunctionPtr)((void*)Bar);
  }
  
  int main(int argc, char *argv[]) {
  	Bar((void*)Bar);	
  }

The inliner will recurse infinitely because it doesn't handle the case where a function takes a function pointer argument, and is called using a pointer to itself as an argument. This patch makes the inliner quit when the examined callsite involves a function pointer parameter.


https://reviews.llvm.org/D41719

Files:
  lib/Analysis/InlineCost.cpp


Index: lib/Analysis/InlineCost.cpp
===================================================================
--- lib/Analysis/InlineCost.cpp
+++ lib/Analysis/InlineCost.cpp
@@ -1254,6 +1254,17 @@
     return Base::visitCallSite(CS);
   }
 
+  // Check if the callee takes a function pointer argument. If it does, then the
+  // caller could be passed to the callee, so we should back out.
+  for (Use &U : CS.getInstruction()->operands()) {
+    if (CastInst *CI = dyn_cast<CastInst>(U)) {
+      Type *DestTy = CI->getDestTy();
+      if (DestTy->isPointerTy() &&
+          DestTy->getPointerElementType()->isFunctionTy()) 
+        return false;
+    }
+  }
+
   // Otherwise we're in a very special case -- an indirect function call. See
   // if we can be particularly clever about this.
   Value *Callee = CS.getCalledValue();


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D41719.128564.patch
Type: text/x-patch
Size: 827 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20180103/dcdaca4f/attachment.bin>