[LLVMdev] SAFECode testsuite query

[John Criswell]

On 2/26/15 9:54 AM, Jyoti Rajendra Allur wrote:
> Hello All,
> I am looking at exploring what benefits SAFECode has to offer over clang S.A and llvm's instrumentation tools like memory sanitizer and address sanitizer.

Are you looking for an off-the-shelf tool, or are you looking for 
approaches to use in your own tool?

> I could come up with the following that are not provided in ASAN/MSAN/Clang S.A
> -> dangling pointer error and detection

First, Clang's static analyzer is a static analysis bug finding tool.  
It won't be able to find every memory safety error in a program.  In 
contrast, tools like Address Sanitzer and SAFECode instrument programs 
to detect (or mitigate) memory safety errors at run-time.

Second, focusing on tools like ASan, SAFECode, etc., a more accurate 
statement is that:

a) SAFECode, with automatic pool allocation enabled, can make dangling 
pointers "harmless" (in that they will either be detected or will not 
cause a type-safety violation, as defined by DSA's type inference results).

b) The SoftBound + CETS system (an option currently built into SAFECode) 
can detect dangling pointer errors (but with much higher overhead).

Please note that automatic pool allocation is currently disabled by default.

> -> crashes in system libraries due to security vulnerabilities.

SAFECode augments or replaces various C library functions with 
memory-safe equivalents (that use the SAFECode metadata to do run-time 
checks).  I would think Address Sanitizer (ASan) would have that as 
well, but I'm not certain.  If it doesn't, I'm sure it could be added.

A third benefit of SAFECode and SoftBound is that they do real bounds 
checking.  Last I checked, ASan spaces memory objects out so that there 
is unused space between them; it then uses load and store checks which 
will detect a bounds violation if it reads or writes invalid memory.  
The ASan approach is technically unreliable; it is possible for a 
pointer to "jump" from one memory object to another undetected.  In 
contrast, SAFECode and SoftBound add run-time checks to pointer 
arithmetic operations, so they know when pointers go out of bounds.  
That said, I'm not sure if ASan's bounds checking limitation is an issue 
in practice.

A fourth benefit is that, in my opinion, SAFECode's debugging output is 
easier to read.  However, since I wrote it, I'm a bit biased. :)

One advantage ASan has over SoftBound and SAFECode is that it is 
integrated into LLVM and has industry developers dedicated to it. 
SAFECode and SoftBound are currently maintained by research groups.

> In the process, I wanted to run the testsuite of safecode and poolalloc but could not find any documentation about how to run those testsuite. It would be great if someone pointed that out for me.

For the lit tests, cd into the test directory and type "make lit". These 
tests verify basic functionality and run some buffer overflow benchmark 
tests on SAFECode.

To run SAFECode on the LLVM test suite, I believe the progdebug target 
is the one you want to use.

> Also, were there any design scalability issues due to which SAFECode has not been supported beyond llvm 3.2 ?
> If yes, could someone let me know about them?

There is no fundamental reason why SAFECode and SoftBound cannot work 
with newer versions of LLVM.  Someone just needs to take the time to 
update the code to use the new LLVM C++ API.

Regards,

John Criswell

-- 
John Criswell
Assistant Professor
Department of Computer Science, University of Rochester
http://www.cs.rochester.edu/u/criswell