[llvm-dev] RFC: callee saved register verifier

[Sanjoy Das via llvm-dev]

Hi,

This is a proposal to introduce a mechanism to LLVM that uses runtime
instrumentation to verify that a call target preserves all of the
registers it says it does.  Internally, we have a diverse set of
calling conventions for runtime functions that get called from LLVM
compiled code, and having some sort of validation mechanism will give
us more confidence in selecting aggressive calling conventions for
these.

The general idea is this: before every call store a cookie in all the
registers that the call is supposed to preserve, and after the call
returns, assert that the supposedly callee saved registers weren't
clobbered.  So for a call target that respects the C calling
convention, we'll generate something like:

   movabsq $0xCA5FCA5FCA5FCA5F, %rbp
   movabsq $0xCA5FCA5FCA5FCA5F, %rbx  # can also be movq %rbp, %rbx etc.
   movabsq $0xCA5FCA5FCA5FCA5F, %r12
   movabsq $0xCA5FCA5FCA5FCA5F, %r13
   movabsq $0xCA5FCA5FCA5FCA5F, %r14
   movabsq $0xCA5FCA5FCA5FCA5F, %r15
   callq   foo
   movabsq $0xCA5FCA5FCA5FCA5F, %rax
   cmpq    %rax, %rbp
   jne     .LBB1_5
   movabsq $0xCA5FCA5FCA5FCA5F, %rax
   cmpq    %rax, %rbx
   jne     .LBB1_5
   movabsq $0xCA5FCA5FCA5FCA5F, %rax
   cmpq    %rax, %r12
   jne     .LBB1_5
   movabsq $0xCA5FCA5FCA5FCA5F, %rax
   cmpq    %rax, %r13
   jne     .LBB1_5
   movabsq $0xCA5FCA5FCA5FCA5F, %rax
   cmpq    %rax, %r14
   jne     .LBB1_5
   movabsq $0xCA5FCA5FCA5FCA5F, %rax
   cmpq    %rax, %r15
   jne     .LBB1_5


The 0xCA5FCA5FCA5FCA5F cookie was selected arbitrarily, but in general
it can be anything that the call target is unlikely to clobber a CSR
with (so it shouldn't be 0, 1, -1 etc.).

This isn't intended to be fast, but is something you turn on in a slow
"validate corner cases" mode.  I think for this to be practical we'll
need to be able to switch this on and off per calling convention,
since I suspect doing this for every call will be impractically slow,
even for a validation run.

Does the general idea sound reasonable?

Note: the focus here is **not** verifying the register allocator or
any other part of _LLVM_, but to verify that e.g. a custom calling
convention that says that a call target preserves %r10 actually does
preserve %r10.  This is the compiler enforcing a contract, just at a
very low level.


Implementation wise, I have a rough prototype that works by running a
MachineFunctionPass before register allocation that emits the
necessary MachineInstrs / control flow to make this happen.  Is that a
right way to solve the problem?  If so, I'll clean up what I have and
put it up for review on llvm-commits.  If there are better ways to do
this, then I'm happy to hear them.

-- Sanjoy