[PATCH] D41719: [InlineCost] Prevent infinite recursion on function pointers

[Jessica Paquette via llvm-commits]

Restricting the iteration count/recursion depth is a great idea, so I’m going to move forward with that.

I’ll update the patch once I’ve finished that (and remember to attach a test case this time, whoops).

- Jessica

> On Jan 4, 2018, at 7:50 AM, Davide Italiano <davide.italiano at gmail.com> wrote:
> 
> On Wed, Jan 3, 2018 at 3:36 PM, Eli Friedman via Phabricator via
> llvm-commits <llvm-commits at lists.llvm.org> wrote:
>> efriedma added a comment.
>> 
>> The bug that's getting triggered by the testcase is that CallAnalyzer::analyzeCall is recursive, without any recursion limit, so it crashes by overflowing the stack.
>> 
>> Your patch doesn't solve that issue in general; it only solves the problem for your specific testcase.  You could trigger a similar crash with a program that isn't recursive.  Or there might be some other way to trick CallAnalyzer into following a recursive program.  If you want to actually fix the bug, the solution is to either add a recursion depth limit, or make the algorithm iterative and add an iteration limit.
>> 
>> 
> 
> Replying here as HN took down Phabricator. I agree with Eli this is
> the correct solution. I have a mild preference for moving the
> algorithm to be iterative, FWIW.
> 
> Thanks,
> 
> --
> Davide