[PATCH] Implement function type checker for the undefined behavior sanitizer.

[Peter Collingbourne]

  What about the compiler-rt changes?


================
Comment at: lib/CodeGen/CGExpr.cpp:3128
@@ +3127,3 @@
+  if (getLangOpts().CPlusPlus && SanOpts->Function &&
+      (!TargetDecl || !isa<FunctionDecl>(TargetDecl))) {
+    if (llvm::Constant *PrefixSig =
----------------
Richard Smith wrote:
> It'd be nice if we could erase these checks if the optimizer resolves the call.
Sure.  A design point of function prefix data is that it permits this sort of optimization.  Maybe if I get a chance I'll see if I can teach the optimizer to do this.

================
Comment at: lib/CodeGen/TargetInfo.cpp:607-608
@@ +606,4 @@
+                   (0x06 << 8) |  //           .+0x08
+                   ('F' << 16) |
+                   ('T' << 24);
+    return llvm::ConstantInt::get(CGM.Int32Ty, Sig);
----------------
Richard Smith wrote:
> Peter Collingbourne wrote:
> > Richard Smith wrote:
> > > What do 'F' and 'T' demangle as? An invalid instruction encoding would give me more confidence here.
> > "rex.RX push %rsp", according to objdump.  But I don't think it really matters what this decodes to.  If the instruction pointer finds itself here the situation isn't dissimilar to jumping into the middle of a multibyte instruction.  (Besides, the RTTI pointer could decode to anything.)
> I think it matters: it's not unreasonable for a function to start as:
> 
>     jmp +8
>     6 byte loop body
>     check loop condition
>     conditional jmp back to loop body
> 
> IIUC, your check would flag a false positive on indirect calls to such a function.
> 
> However, since the `'F'` is an instruction prefix that is redundant on a `'T'` (`pushq %rsp`) instruction, I think it's reasonable to assume that the `FT` signature will not occur in uninstrumented code.
I see.  I think I agree with your reasoning.


http://llvm-reviews.chandlerc.com/D1338

BRANCH
  ubsan-function

ARCANIST PROJECT
  clang