[stackprotector] Add the llvm.stackprotectorcheck intrinsic

[Michael Gottesman]

The attached patch adds the llvm.stackprotectorcheck intrinsic. First I am going to describe the problem the intrinsic is meant to solve and how it fits into said problem's final solution. I am fine with any comments about the general algorithm being in response to this.

*NOTE* I was purposely trying to not rewrite the stack protector pass itself.
*NOTE* The following analysis ignores OSes which do not support the normal LLVM stack protector check implementation (i.e. OpenBSD). In such cases, the normal stack protector check pass will still be used.

==========
The Problem
==========

For those unfamiliar, a sibling call is a specific type of safe tail call optimization which LLVM automatically recognizes/performs. Currently the stack protector pass disrupts sibling calls since we insert the stack protector check calls before we lower calls from LLVM IR -> SelectionDAG nodes where the decision is made upon whether or not a sibling call optimization is safe. Thus if originally we had the following IR:

----
%x = tail call i8* @cool_function()
ret %x
----

The stack protector pass will modify said code like so:
----
%x = tail call i8* @fun()
%guard = load …
%stackslot = load …
%cmp = icmp i8* %guard, %stackslot
br i1 %cmp, label %success, label %failure

success:
  ret %x

fail:
  call void @__stack_chk_fail()
  unreachable.
----

Since “ fun” is no longer in the “tail position”, no sibling call can be generated when we lower the LLVM IR Call to a selection dag node. Notice that if we were able to speculate and realize that the sibling call would actually occur without the stack protector, since we are reusing the stack, it would be safe to commute the call to cool_function and the stack protector, i.e.:

----
%guard = load …
%stackslot = load …
%cmp = icmp i8* %guard, %stackslot
br i1 %cmp, label %success, label %failure

success:
  %x = tail call i8* @fun()
  ret %x

fail:
  call void @__stack_chk_fail()
  unreachable.
----

giving us the tail call *AND* the protection of the stack protector. Sadly we can not recognize if “fun” will actually tail call at the IR level since on certain platforms the sibling call decision requires target dependent knowledge of which registers are in use (see X86ISelLowering::IsEligibleForTailCallOptimization). Thus a different scheme is needed.

==========
The Solution
==========

The key realization that occurred to me is that at the MI level tail calls are represented as a form of special return statement. If we could delay the expansion of the stack protector check, we could just allow for normal CodeGen to occur and always insert the stack protector check (and the branches to its basic blocks) right before the return statement.

Thus consider the following solution to the stated problem:

1. Introduce an llvm.stackprotector check intrinsic (this patch). This represents to code-gen the comparison, branch, and two basic blocks/etc. Have the stack protector pass insert it as appropriate. Thus at the IR level one would see:

----
%x = tail call i8* @fun()
%guard = load …
%stackslot = load …
call void @llvm.stackprotectorcheck(i8* %guard, i8* %stackslot)
ret %x
----

2. If in the stack protector pass we can prove that there is a call right before the return statement which satisfies all the platform independent requirements for a tail call, we swap the order of the stack protector and the call, i.e.:

----
%guard = load …
%stackslot = load …
call void @llvm.stackprotectorcheck(i8* %guard, i8* %stackslot)
%x = tail call i8* @fun()
ret %x
----

3. In SelectionDAGBuilder, when we see the llvm.stackprotectorcheck intrinsic we perform a similar operation to what we do with switch statements and other cases where we need to generate new MachineBasicBlocks for an instruction:
	a. When we visit the actual llvm.stackprotectorcheck intrinsic, we create success/failure successor machine basic blocks for the current machine basic block and initialize a StackProtectorDescriptor structure on SelectionDAGBuilder with a reference to the original machine basic block, the two new basic blocks, and the llvm IR llvm.stackprotectorcheck intrinsic. We export from the current basic block both load operands.
	b. SelectionDAGISel visits the end of the basic block and makes the decision on whether or not the tail call will occur.
	c. In SelectionDAGISel::FinishBasicBlock:
		i. Split the parent machine basic block at its first terminator (which will be either a tail call return MI or a return MI) and put the back part of the split parent MBB into the success MBB using MachineBasicBlock::splice.
		ii. Use Selection DAG builder to generate the comparison/new branch instructions for the end of the Parent MBB and the entirety of the failure MBB.

Please review with any flames/comments/suggestions/etc,

Michael

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-stackprotector-Added-intrinsic-llvm.stackprotectorch.patch
Type: application/octet-stream
Size: 3770 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20130723/202d3abd/attachment.obj>