[PATCH] D12923: Add support for function attribute "notail"

[Akira Hatanaka via llvm-commits]

ahatanak added a comment.

In http://reviews.llvm.org/D12923#247641, @sanjoy wrote:

> In http://reviews.llvm.org/D12923#247639, @ahatanak wrote:
>
> > No, tail call optimization doesn't have to be disabled on indirect calls.
>
>
> What if the indirect callee is marked with `notail`?  Isn't the whole point of `notail` that functions marked `notail` won't have tail calls to them?


Yes, that's correct. We want to avoid doing tail call optimization on a call site if the compiler knows it is a call to a notail function. If the call is an indirect call, it's not always possible to know that.

> Perhaps I'm missing something here, so it helps to be concrete -- in the C example I gave, what's the expected behavior of the program if the function pointer passed to `f` via `ptr` is a `notail` function?


It should have no effect. If the compiler cannot determine a call site is a call to a nontail function, it doesn't block tail call optimization.

> I think the `notail` should be on the call instruction, not on the callee; as some sort of "inverse" of `musttail`.  Then we can teach the optimizer to never codegen a `notail` call as a tail call.


I think attaching notail to the call instruction is more limiting than attaching it to the callee function in some cases. Suppose we are compiling a code like this with -O3:

int f(g* ptr, int x) {

  return ptr(ptr, x);

}

int __attribute((notail)) foo2(void*, int);

int foo1(int a) {

  return f(foo2, a);

}

After all the optimization passes (including the inliner) are run, the IR will look like this:

define i32 @foo1(i32 %a) #0 {
entry:

  %call.i = call i32 @foo2(i8* bitcast (i32 (i8*, i32)* @foo2 to i8*), i32 %a) #2
  ret i32 %call.i

}

If the notail attribute was on the call instruction, there would be no way to tell tail call optimization shouldn't be done on the call to foo2.


http://reviews.llvm.org/D12923