[llvm-dev] [PATCH] D12923: Add support for function attribute "notail"

[Sanjoy Das via llvm-dev]

 > That was what I had in mind: the function attribute blocks tail call for statically direct calls but doesn't promise
 > anything (in fact, does nothing) for indirect calls.
 >
 > Do you think we shouldn't make any promises for statically direct calls either? I don't see why it's hard to keep the
 > promise that direct tail calls will be blocked. Do you have a particular optimization in mind that would be difficult or
 > impossible to implement if we promised to block direct calls?

I think in this scheme we'll have problems around devirtualization.
For instance, you could start with a indirect call to a notail target
that would get TCO'ed (as an indirect call), after which
devirtualization or load/store motion would turn the indirect call
into a direct call; and you'd end up with a direct tail call to a
notail target.

I don't know what your requirements are, but if I were you I'd design
the `notail` marker to be logically part of a function's calling
convention.  That way functions marked `notail` get to do things that
are legal only if the call to them wasn't a tail call, because all
well-defined calls constrain the caller to match callee's expectation
with respect to tail calls.  In the CFE, you could make the `notail`
attribute have the same restrictions / conventions as things like
`fastcall`.

-- Sanjoy