[llvm-commits] [llvm] r67849 - /llvm/trunk/test/CodeGen/PowerPC/int-fp-conv-0.ll

[Chris Lattner]

On Apr 1, 2009, at 1:33 PM, Misha Brukman wrote:
> On Tue, Mar 31, 2009 at 9:04 PM, Chris Lattner <clattner at apple.com>  
> wrote:
> On Mar 30, 2009, at 1:34 PM, Misha Brukman wrote:
>> I'm proposing we test passes in C++ instead of a shell script that  
>> calls grep.
>
> What is the real value of doing this?  The costs I see are 1)  
> increased link time and 2) more dependence on the API (more to  
> update when an API changes).
>
> I address your issue (1) in #3 below.  I accept your point (2) as  
> valid, but based on personal experience, although it's a non-zero  
> effort, it's not as painful as you imagine, and coupled with the  
> benefits below, I think it's worthwhile.

> Consider it a tradeoff (of CPU time for human time), rather than  
> simply an added cost.

I don't agree with this at all.  I spend a huge amount of time waiting  
for builds and make check to complete.  Machine time is human time.

> Let's look at the benefits:
>
> 1) Tests can be written at any level of the LLVM API, so you can  
> write tests to verify a single function, check that a class modifies  
> its state properly (even private state), verify a pass output, or  
> verify that llc or opt does everything end-to-end correctly.  For  
> instance, let's say we have a transformation that's dependent on an  
> analysis, and say we have a test case where the transformation  
> should have no effect.

While I agree that there is minor value here, I don't agree that this  
is a big win.  Testing is a set of tradeoffs.  We want to make it easy  
for people to write tests (because we want them to actually *do* it)  
and we want the tests to be as reasonably stable across future changes  
as possible to reduce maintenance.  We also don't want testing to take  
an unacceptable amount of time, otherwise people won't run them  
regularly.  We accept that testing will never catch all bugs, but  
that's ok, it is just one weapon in the arsenal of quality.

> We could test with the opt/diff to make sure the output is the same  
> as input, but that's not really testing what we think we are,  
> because it's testing the interaction of the two, so a bug in one can  
> mask a bug in another.  In other words, there are many code paths  
> that can "return false" along the way, and cause the transformation  
> not to be applied, but do you really know WHICH one is responsible?

Who cares?  Why is it so important to test that level of detail?  LLVM  
as a whole has *very* strong APIs and interfaces, the most important  
of which is the IR itself.  Anything that operates on the IR is going  
to be very stable, and all mid-level optzn passes should be easily  
observable based on what they do.

> With a C++ test, you can test the analysis pass separately and test  
> its outputs on one test, and then in another test, inject a Mock (or  
> Fake) analysis object with known values, and see how the pass  
> behaves.  This lets you test each piece of code independently, and  
> prevents complex interactions from messing with your tests.

We can do this without a c++ test, just have clients like -aa-eval and  
other things that drive the analysis in certain ways.

> 2) Although LLVM may have high line-coverage via large tests, we  
> don't know what code path coverage it has, which is also important.   
> If we can only test at the granularity of a pass, there are 2**N  
> code paths, and it's hard to contrive .ll inputs that will cause  
> function foo() to be called with inputs bar and quux (also makes  
> debugging more difficult than it should be, see #3 below).  With a  
> single test, you can synthesize exactly what you want the inputs to  
> be, thus avoiding single-stepping through a debugger when you don't  
> need to.  So instead of writing, say O(2**N) tests, you can write  
> O(N) tests (in number, not runtime!) to get much better path coverage.

I don't see this to be as important as you do :)

> 3) Currently, when something is broken, bugpoint is introduced to  
> narrow down the test case to something small, and then the developer  
> steps through the code with a debugger will (hopefully) to find  
> which part of which data structure isn't being updated properly.  I  
> have personally been spending *MUCH* less time interacting with  
> debuggers thanks to testing -- instead of setting a break point and  
> manually verifying state at function entry/exit, I can set up the  
> test scenario, and call EXPECT() to verify what I think it should  
> be, and let it run!  I'll quickly know if it's a problem or a red  
> herring.  This works for any granularity of code, and is really  
> valuable.
>
> Without tests, each developer manually repeats the debugging via  
> "Oh, I think it's caused when X happens and Y interacts with Z!".   
> If you have a test for that that you commit to the repository,  
> anyone who thinks it's broken can just re-run the test and see that  
> no, that guess was wrong, try again.

As you know Misha, LLVM has very strong interfaces.  We have  
guarantees like "no pass should crash on a valid IR input".  This is  
why things like bugpoint work.  I personally have not suffered from  
the debugger issues that you are describing.

Most of the bugs that I have spent a lot of time scratching my head  
over are not logic bugs in the pass, they are things like dangling  
pointer and other fun bugs.  The most insidious one is the "dangling  
pointer as the key of a map" which manifest non-determinstically when  
malloc happens to recycle a node, making a new node "already be in the  
map".  This is a great reason to improve the LLVM infrastructure (e.g.  
the new AssertingVH<> class) to help make these bugs impossible.  This  
would not have been caught with any reasonable testing.

> 4) There are other things we cannot test via opt/grep directly, or  
> even llvmirgrep:
> (a) anything that doesn't have text-based output to grep through, e.g.
>   (i) analysis passes (alias analysis, etc.)

These have text output with --aa-eval and friends.  This is obviously  
true  because we have many basicaa tests already.

>   (ii) PassManager (e.g., verify pass ordering in the presence of  
> FunctionPasses and ModulePasses, etc. that's independent of code  
> being run)

-debug-pass=Arguments / Structure still work :)

> (b) things that read or emit data are only tested indirectly:  
> bitcode reader/writer,

We have plenty of tests for these.

> TableGen, code emitters, etc.

These get tested by whether or not the compiler works.

> Bottom line -- I see at least 2 open bugs on Andersen's that would  
> benefit.

Andersen's would benefit from contributors who care about it, not more  
tests :)

>   How about we expose its class in a header to be able to write  
> tests and see how it goes?  And there is a bug for a pass that seems  
> to run out of memory that might also benefit.  And next time there's  
> some difficult-to-debug problem that you see as similar to the  
> above, please file a bug and assign it to me, I'll see what I can do  
> for it with some targetted tests.  How does that sound?

No.  Why does andersen's need any change to it to make it more  
testable?  What part of its public interface is not testable?

>
>> How else can we check the correctness of a pass, which involves  
>> pattern-matching multiple instructions?  Grep can only look at a  
>> single line at a time, and doesn't have the semantic knowledge of  
>> what's an instruction and what's an operand, whereas we have all  
>> this information in C++.
>
> I think the answer is that we need to build *one* small "llvmirgrep"  
> tool that does this, which could be used by many different tests.
>
> This is a tool that doesn't exist yet, which means it will require  
> time and effort to design and implement it, as well as (dare I say  
> it?) test it.

Of course.  This is an infrastructure investment to make future  
testing easier.   Having llvm-irgrep would make it much easier to  
write tests (increasing the odds that they will actually be written)  
than having to write C++ code. In case you haven't noticed, C++ is a  
very low-level language, and not one that is awesome at pattern  
matching ;-)

-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20090401/8da4177c/attachment.html>