[LLVMdev] SAFECode testsuite query

[Kostya Serebryany]

On Thu, Feb 26, 2015 at 8:49 AM, John Criswell <jtcriswel at gmail.com> wrote:

> On 2/26/15 9:54 AM, Jyoti Rajendra Allur wrote:
>
>> Hello All,
>> I am looking at exploring what benefits SAFECode has to offer over clang
>> S.A and llvm's instrumentation tools like memory sanitizer and address
>> sanitizer.
>>
>
> Are you looking for an off-the-shelf tool, or are you looking for
> approaches to use in your own tool?
>
>  I could come up with the following that are not provided in
>> ASAN/MSAN/Clang S.A
>> -> dangling pointer error and detection
>>
>
> First, Clang's static analyzer is a static analysis bug finding tool.  It
> won't be able to find every memory safety error in a program.  In contrast,
> tools like Address Sanitzer and SAFECode instrument programs to detect (or
> mitigate) memory safety errors at run-time.
>
> Second, focusing on tools like ASan, SAFECode, etc., a more accurate
> statement is that:
>
> a) SAFECode, with automatic pool allocation enabled, can make dangling
> pointers "harmless" (in that they will either be detected or will not cause
> a type-safety violation, as defined by DSA's type inference results).
>
> b) The SoftBound + CETS system (an option currently built into SAFECode)
> can detect dangling pointer errors (but with much higher overhead).
>
> Please note that automatic pool allocation is currently disabled by
> default.
>
>  -> crashes in system libraries due to security vulnerabilities.
>>
>
> SAFECode augments or replaces various C library functions with memory-safe
> equivalents (that use the SAFECode metadata to do run-time checks).  I
> would think Address Sanitizer (ASan) would have that as well, but I'm not
> certain.  If it doesn't, I'm sure it could be added.
>
>
asan replaces a whole bunch of libc functions so that we find bugs where an
invalid pointer is passed to libc.

This is still not perfect. We also want to find bugs in the system
libraries themselves,
and we've been advocating for instrument-the-world approach.
See e.g.
http://www.chromium.org/developers/testing/instrumented-libraries-for-dynamic-tools

There are scary things lurking in the system libraries.



> A third benefit of SAFECode and SoftBound is that they do real bounds
> checking.  Last I checked, ASan spaces memory objects out so that there is
> unused space between them; it then uses load and store checks which will
> detect a bounds violation if it reads or writes invalid memory.  The ASan
> approach is technically unreliable; it is possible for a pointer to "jump"
> from one memory object to another undetected.


Correct. This is a speed-vs-accuracy-vs-complexity trade-off


> In contrast, SAFECode and SoftBound add run-time checks to pointer
> arithmetic operations, so they know when pointers go out of bounds.  That
> said, I'm not sure if ASan's bounds checking limitation is an issue in
> practice.
>
> Depending on the definition of "issue in practice".
asan finds lots of bugs everywhere we tried,
but it is by no means a verification tool.
Neither is any other *dynamic* testing tool for C++ that I am aware of.


> A fourth benefit is that, in my opinion, SAFECode's debugging output is
> easier to read.  However, since I wrote it, I'm a bit biased. :)
>
> One advantage ASan has over SoftBound and SAFECode is that it is
> integrated into LLVM and has industry developers dedicated to it. SAFECode
> and SoftBound are currently maintained by research groups.
>
>  In the process, I wanted to run the testsuite of safecode and poolalloc
>> but could not find any documentation about how to run those testsuite. It
>> would be great if someone pointed that out for me.
>>
>
> For the lit tests, cd into the test directory and type "make lit". These
> tests verify basic functionality and run some buffer overflow benchmark
> tests on SAFECode.
>
> To run SAFECode on the LLVM test suite, I believe the progdebug target is
> the one you want to use.
>
>  Also, were there any design scalability issues due to which SAFECode has
>> not been supported beyond llvm 3.2 ?
>> If yes, could someone let me know about them?
>>
>
> There is no fundamental reason why SAFECode and SoftBound cannot work with
> newer versions of LLVM.  Someone just needs to take the time to update the
> code to use the new LLVM C++ API.
>
> Regards,
>
> John Criswell
>
> --
> John Criswell
> Assistant Professor
> Department of Computer Science, University of Rochester
> http://www.cs.rochester.edu/u/criswell
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150226/2ec48cca/attachment.html>