[PATCH] Implement function type checker for the undefined behavior sanitizer.

[Richard Smith]

  LGTM


================
Comment at: lib/CodeGen/TargetInfo.cpp:607-608
@@ +606,4 @@
+                   (0x06 << 8) |  //           .+0x08
+                   ('F' << 16) |
+                   ('T' << 24);
+    return llvm::ConstantInt::get(CGM.Int32Ty, Sig);
----------------
Peter Collingbourne wrote:
> Richard Smith wrote:
> > What do 'F' and 'T' demangle as? An invalid instruction encoding would give me more confidence here.
> "rex.RX push %rsp", according to objdump.  But I don't think it really matters what this decodes to.  If the instruction pointer finds itself here the situation isn't dissimilar to jumping into the middle of a multibyte instruction.  (Besides, the RTTI pointer could decode to anything.)
I think it matters: it's not unreasonable for a function to start as:

    jmp +8
    6 byte loop body
    check loop condition
    conditional jmp back to loop body

IIUC, your check would flag a false positive on indirect calls to such a function.

However, since the `'F'` is an instruction prefix that is redundant on a `'T'` (`pushq %rsp`) instruction, I think it's reasonable to assume that the `FT` signature will not occur in uninstrumented code.

================
Comment at: lib/CodeGen/CGExpr.cpp:3128
@@ +3127,3 @@
+  if (getLangOpts().CPlusPlus && SanOpts->Function &&
+      (!TargetDecl || !isa<FunctionDecl>(TargetDecl))) {
+    if (llvm::Constant *PrefixSig =
----------------
It'd be nice if we could erase these checks if the optimizer resolves the call.


http://llvm-reviews.chandlerc.com/D1338

BRANCH
  ubsan-function

ARCANIST PROJECT
  clang