[cfe-dev] [RFC] Extending and improving Clang's undefined behavior checking

[Chris Lattner]

<catching up on old email>
On Aug 10, 2012, at 7:48 PM, Richard Smith <richard at metafoo.co.uk> wrote:
> There are three different (and mostly orthogonal, design-wise) areas where I would like to make improvements to Clang's undefined behavior checking:

This is truly awesome, I would really love to see this.  Your design proposal is somewhat startlingly similar to an approach I pitched to John and Vikram sometime last year. :-)  I really like separating out the orthogonal pieces and pulling together the various disparate pieces into something coherent.

> 1) Completeness of checks. There are integer undefined behaviors which -ftrapv and -fcatch-undefined-behavior don't catch, and there is almost no checking available for any other undefined behaviors outside of those and the ones caught by {Address,Thread,Memory} Sanitizer.

Yes.  Nuno's bounds checking work can also be pulled into this eventually, as could stack canaries and "fortify source".

> 2) Command-line interface. We currently have the following options to enable various flavors of undefined 

> I would like for us to have a single argument which enables all undefined behavior checking which can be performed cheaply and with no changes to address space layout or ABI; I propose we extend -fcatch-undefined-behavior to be that argument.

+1

> I think we should support this kind of configuration through a mechanism similar to warning flags: -fcatch-undefined-behavior=c++11 -fno-catch-undefined-behavior=null-reference, for instance.

+1

> Also, I think we should consider renaming this switch (and/or possibly the -f*-sanitizer switches) to provide a consistent set of names, but I don't have a concrete proposal for that.

That would make sense to me.

> 3) Handling and reporting of undefined behavior, once caught. The sanitizers produce decent information (which is both useful and detailed), but it could be prettier. The other checks just emit @llvm.trap(), which is far from ideal.

Yes, this should be orthogonal from the checks and interface, and should be consistent across all the checks.

> IOC is in some ways quite ambitious here, and provides options to not only diagnose a problem, but also to call a function which can produce a value for the operation and recover. I'm not proposing adding such a feature to Clang at this time (I'll leave that to the IOC folks).

I agree.

> I would like to propose adding a runtime library for the -fcatch-undefined-behavior checks, and emit calls to relevant diagnostic functions instead of calls to @llvm.trap. I'm planning on having one library entry point per flavor of check, named __ubsan_report_<check_name>, which would be passed relevant operand values, and a pointer to a structure containing information about the context (file, line and column number, operand types as strings, and possibly a code snippet to display with the diagnostic).
> 
> As a strawman proposal for runtime error output, we could aim to mirror the formatting style of compile-time diagnostics (since we've already got one way of beautifully presenting diagnostic information, why invent another?).

This would be pretty cool.

> This proposal intends to incorporate some, but not all, of IOC into mainline Clang. The parts which I'm not currently intending to address as part of this work are:
>  * -fuse-intrinsic. This allows the choice of whether to use LLVM's overflow intrinsics. I intend to use such intrinsics unconditionally.
>  * -fuse-random-value. This allows recovery from detected undefined behavior by providing a value for the operation, which is not part of my proposal.
>  * -fhandler-null. This selects the usage of an alternate runtime library.
>  * -fchecks-num. This provides logging output when checks are inserted, and would not be valuable to my target audience.
>  * -fcatch-non-ubc-type. This is designed to catch unsigned integer overflow, and some other cases which don't have undefined behavior, and so is not directly connected to this work.

Makes sense to me.  I'd like to have these checks be simplified down, even if it means that we lose some fine-grained control.  I'd really like to get to the point where we can recommend people to turn on -fcatch-undefined-behavior (thus getting some sane subset of checks) when they see strange behavior.  

-Chris