[cfe-dev] A tool for generating reduced headers for test cases (delta debugging)

[Ted Kremenek]

On Jul 9, 2008, at 9:16 AM, Eli Friedman wrote:

> On Wed, Jul 9, 2008 at 1:49 AM, Nikita Zhuk <nikita at zhuk.fi> wrote:
>> Hi,
>>
>> Several clang analysis test cases written in ObjC contain the
>> following comment line in their header: "These declarations were
>> reduced using Delta-Debugging from Foundation.h on Mac OS X.". Are
>> there some instructions about tools which should be used to generate
>> such reduced headers?
>
> If I'm not mistaken, the steps are just to generate a full file with
> clang -E, then reduce the file using Delta (http://delta.tigris.org/).
> But Ted should be able to comment with more complete steps.

That's exactly right.

The Delta tool requires a user-provided script that discerns a "test  
case" between good and bad, and tries to remove pieces from the test  
case (which is any text file) until a minimal test case is generated  
that is still "good".

In the case of header file reduction, I'm working with two files:

t.i   (the preprocessed header goop that I want to reduce)
t.c  (the file that includes t.i)

The "test case" is t.i, and the my script (called "detect_error.pl")  
classifies a version of t.i as good or bad by doing the following:

1) If "gcc -fsyntax-only t.c" fails, the test case is BAD.
2) if "clang -fsyntax-only t.c" fails, the case is BAD.
3) Otherwise, the test case is GOOD.

The trick is identifying what is failure: gcc emiting warnings, gcc or  
clang crashing, etc.

Delta-debugging then whittles down t.i until it cannot make it any  
smaller.  Once it reduces t.i, I just include the contents of t.i  
inside t.c (or t.m, or whatever).

>> Being able to do so would make it possible to
>> report bugs by using runnable test cases, and those test cases could
>> be eventually included with the clang itself.

I can certainly clean up the detect_error.pl script and post it the  
website with directions.

> Reducing out Foundation.h only really helps for non-runnable
> testcases; the presence of Foundation.h is a pretty good indicator of
> the presence of the Foundation classes. :)

Exactly.  In general, header reduction might be good for filing any  
kind of test case against Clang that depends on system/library- 
specific APIs.