[cfe-dev] clang feature request: integer overflow check in operator new

[Felix von Leitner]

Thus spake David Chisnall (theraven at sucs.org):
> > I have a humble feature request for clang, now that it officially
> > supports C++: integer overflow check in operator new.
> > 
> > Basically, in code like this:
> > 
> >   int* foo=new int[somevalue];
> > 
> > the compiler does an implicit somevalue*sizeof(int) and passes that
> > value to operator new.  If that multiplication overflows, this is a
> > security vulnerability.  The Microsoft solution is to use the overflow
> > flag on x86 to set the resulting value to (size_t)-1, which will then
> > make operator new fail.  There are caveats to this, obviously, but it is
> > an important step to make and it cannot be done without compiler help.
> This looks like it should be relatively easy to add.  Line 444 of CGExprCXX.cpp just needs changing to emit a call to the overflow-checking intrinsic (see EmitOverflowCheckedBinOp() in CGExprScalar.cpp), and then a little extra handling for the case where it overflows.  I'm not sure what the effect of passing (size_t)max to new in GNU libstdc++ is - perhaps the best thing would be to bypass the new entirely and have it return 0 in overflow?

operator new is supposed to throw an exception when it fails to allocate
memory, not return NULL.  But you can call operator new in a way that
tells it to return NULL, too.

So some caution is required to get this right :-)

Thanks,
Felix