[LLVMdev] [PROPOSAL] Adding support for -fstack-protector-strong

[Magee, Josh]

Hello,

I plan to implement "Stack Smashing Protection - Strong" support in LLVM.
Below is a description of this feature and an overview of the implementation
plan.  I have divided up the implementation into stages that can be delivered
incrementally.  

I'm looking for any feedback (suggestions, requests, etc) before I actually
begin the work.

Thank you!
Josh


======================================
Stack-protector-strong support in LLVM
======================================

Introduction
------------
Stack Smashing Protection (SSP) aims to protect against buffer overflow attacks
by placing a 'canary' value on the stack that is validated upon exiting a
function.

LLVM currently supports SSP.  Specifically it supports two modes:
  -fstack-protector-all
    - enables stack protectors for all functions
  -fstack-protector
    - Uses a heuristic to apply stack protectors only to functions that need it
    - The heuristic checks for:
      character arrays >= SSP_BUFFER_SIZE (default:8-bytes)
      aggregates containing arrays >= SSP_BUFFER_SIZE *
      alloca >= SSP_BUFFER_SIZE, or variable-sized alloca

These two options/modes are analogous to the options/modes supported by GCC.

* Note, there have been relatively recent commits to LLVM trunk improve the
  state of SSP (e.g., checking for aggregates).


A short proposal was made to add a new SSP option to GCC: -fstack-protector-strong. [1]
Note, I am not the original author of the proposal.  I just want to add it to LLVM :-).
Briefly, this option aims to enable heuristics that provide stronger protection
than -fstack-protector, but avoid the overkill of -fstack-protector-all.  This
option has been accepted into the Google branch of GCC and presumably will
make its way into mainline GCC eventually.

The remainder of this documents discusses stack-protector-strong and presents
the proposed implementation work needed to support it in LLVM.

Requirements
------------
The requirements of -fstack-protector-strong are:
  A) Stack protectors should be enabled for functions when:
    1) An address of a local variable is taken in such a way as to expose the
       address of a stack location.
      - Example: the address of a local on the RHS of an assignment, the
        address of a local passed into function.
      - Note, source level addr-of expressions could be eliminated during
        optimization, in which case they would not expose a stack address.
    2) There is an array, regardless of type or size.
    3) There is an aggregate which contains an array, regardless of array type or size.

  B) The stack should be laid out such that:
    1) Larger arrays and arrays nested in aggregates are closest to the stack guard.
    2) Smaller arrays and arrays nested in aggregates are second closet to the stack guard.
    3) Variables that have their address taken are third closet to the stack guard.
    4) Scalars and pointers are furthest away from the stack guard.


The original proposal [1] lists an additional requirement that stack protectors
should be enabled for functions that have register local variables.  This
requirement seems like it must be an error in the original proposal because it
does not make any sense.  Register-keyword specified variables have no special
effect on CodeGen (the only effect is that the compiler will error if you
attempt to take an address-of such a variable.)  If this was intended to refer
to variables that can be kept entirely in registers for their lifetime then
protectors are also not needed since such a variable would not expose any stack
address.  In fact, the GCC patch does not implement this either, see [2].


Implementation
--------------
The bulk of the work required for stack-protector-strong is already implemented
for the existing stack-protector support.
The proposed work is:
  1 Adding a new IR attribute "sspstrong":
    ssp       - SSP on, use basic heuristic
    sspstrong - SSP on, use strong heuristic
    sspreq    - SSP on, required for function
    - Note, I am proposing the IR attribute because it fits nicely into the
      existing structure.  An alternative choice would be to add a new CodeGen
      option that would cause ssp-specified functions to be analyzed using the
      strong heuristic.  I prefer the attribute because it allows for finer
      granularity (i.e., you could have one CU with both ssp and sspstrong
      specified functions.
    - Initially, sspstrong would work the same as sspreq.  Once the "strong"
      heuristic is implemented, then it would be used for sspstrong.

  2  Adding the necessary command line switches to Clang:
    - -fstack-protector-strong
    - (-cc1) -stack-protector=n
             0 = disabled
             1 = ssp
             2 = sspstrong  [new]
             3 = sspreq
        - Logically, placing sspstrong at index 2 seems most appropriate.
          However, it could be placed at 3 to avoid redefining the existing
          semantics that 2 == sspreq.

  
  3 Add Clang SSP attributes.
    - Clang-level attributes that match the IR attributes and allow ssp,
      sspstrong, and sspreq to be specified on a function-by-function basis.
    - Already implemented privately for ssp and sspreq.  Only sspstrong would
      need to be added.  Ultimately all attributes would be contributed to the
      community.

  4 Implement 'strong' heuristic in CodeGen/StackProtector.cpp
    - Mostly relaxing checks in current StackProtector pass.  For example, when
      strong attribute is present, relax rules requiring arrays of char type,
      ssp-buffer-size, etc.
    - Add analysis to check for taking the address of local variables.
    - Make the IR attribute 'sspstrong' to enable this heuristic.

  5 Implement finer data layout rules.
    - Most support is already there (e.g., large objects assigned before small
      objects).  What is needed is to extend the existing code to allow finer
      granularity and distinction between different SSP-triggering objects.
    - Some minor refactoring desired.  For example, in
      CodeGen/SelectionDAG/FunctionLoweringInfo.cpp, there are checks for
      determining if an object would trigger a stack protectr (MayNeedSP).  These
      checks are inconsistent with the checks in StackProtector.  Both of these spots
      should use a common (i.e., same) check.

  6) Updating/adding documentation.
        

Note that the appropriate regression tests (updating existing tests and adding
new ones) will be included with each stage.


Conclusion
----------
stack-protector-strong support builds upon the existing stack protector
implementation in LLVM.  The primary implementation changes will be extending
analysis to meet the strong requirements and fine tuning the stack layout.  The
most visible change will be the addition of a new IR attribute: sspstrong.


References
----------
[1] Original Proposal: https://docs.google.com/a/google.com/document/d/1xXBH6rRZue4f296vGt9YQcuLVQHeE516stHwt8M9xyU/edit?hl=en_US&ndplr=1&pli=1
[2] Code Review for patch to GCC implenting -fstack-protector-strong: http://codereview.appspot.com/5461043