<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div>On Apr 1, 2009, at 1:33 PM, Misha Brukman wrote:</div><blockquote type="cite">On Tue, Mar 31, 2009 at 9:04 PM, Chris Lattner <span dir="ltr"><<a href="mailto:clattner@apple.com" target="_blank">clattner@apple.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> <div><div><div><div>On Mar 30, 2009, at 1:34 PM, Misha Brukman wrote:</div><blockquote type="cite"><div class="gmail_quote"><div>I'm proposing we test passes in C++ instead of a shell script that calls grep.</div> </div></blockquote></div><div>What is the real value of doing this? The costs I see are 1) increased link time and 2) more dependence on the API (more to update when an API changes).</div></div></div></blockquote><div><br> I address your issue (1) in #3 below. I accept your point (2) as valid, but based on personal experience, although it's a non-zero effort, it's not as painful as you imagine, and coupled with the benefits below, I think it's worthwhile.</div></div></blockquote><div><br></div><blockquote type="cite"><div class="gmail_quote"><div>Consider it a tradeoff (of CPU time for human time), rather than simply an added cost.</div></div></blockquote><div><br></div><div>I don't agree with this at all. I spend a huge amount of time waiting for builds and make check to complete. Machine time is human time.</div><div><br></div><blockquote type="cite"><div class="gmail_quote"><div>Let's look at the benefits:<br><br>1) Tests can be written at any level of the LLVM API, so you can write tests to verify a single function, check that a class modifies its state properly (even private state), verify a pass output, or verify that llc or opt does everything end-to-end correctly. For instance, let's say we have a transformation that's dependent on an analysis, and say we have a test case where the transformation should have no effect.</div></div></blockquote><div><br></div><div>While I agree that there is minor value here, I don't agree that this is a big win. Testing is a set of tradeoffs. We want to make it easy for people to write tests (because we want them to actually *do* it) and we want the tests to be as reasonably stable across future changes as possible to reduce maintenance. We also don't want testing to take an unacceptable amount of time, otherwise people won't run them regularly. We accept that testing will never catch all bugs, but that's ok, it is just one weapon in the arsenal of quality.</div><br><blockquote type="cite"><div class="gmail_quote"><div>We could test with the opt/diff to make sure the output is the same as input, but that's not really testing what we think we are, because it's testing the interaction of the two, so a bug in one can mask a bug in another. In other words, there are many code paths that can "return false" along the way, and cause the transformation not to be applied, but do you really know WHICH one is responsible?</div></div></blockquote><div><br></div><div>Who cares? Why is it so important to test that level of detail? LLVM as a whole has *very* strong APIs and interfaces, the most important of which is the IR itself. Anything that operates on the IR is going to be very stable, and all mid-level optzn passes should be easily observable based on what they do.</div><br><blockquote type="cite"><div class="gmail_quote"><div>With a C++ test, you can test the analysis pass separately and test its outputs on one test, and then in another test, inject a Mock (or Fake) analysis object with known values, and see how the pass behaves. This lets you test each piece of code independently, and prevents complex interactions from messing with your tests.</div></div></blockquote><div><br></div><div>We can do this without a c++ test, just have clients like -aa-eval and other things that drive the analysis in certain ways.</div><br><blockquote type="cite"><div class="gmail_quote"><div>2) Although LLVM may have high line-coverage via large tests, we don't know what code path coverage it has, which is also important. <span class="Apple-style-span" style="-webkit-text-stroke-width: -1; ">If we can only test at the granularity of a pass, there are 2**N code paths, and it's hard to contrive .ll inputs that will cause function foo() to be called with inputs bar and quux (also makes debugging more difficult than it should be, see #3 below). With a single test, you can synthesize exactly what you want the inputs to be, thus avoiding single-stepping through a debugger when you don't need to. So instead of writing, say O(2**N) tests, you can write O(N) tests (in number, not runtime!) to get much better path coverage.</span></div></div></blockquote><div><br></div>I don't see this to be as important as you do :)</div><div><br><blockquote type="cite"><div class="gmail_quote"><div>3) Currently, when something is broken, bugpoint is introduced to narrow down the test case to something small, and then the developer steps through the code with a debugger will (hopefully) to find which part of which data structure isn't being updated properly. I have personally been spending *MUCH* less time interacting with debuggers thanks to testing -- instead of setting a break point and manually verifying state at function entry/exit, I can set up the test scenario, and call EXPECT() to verify what I think it should be, and let it run! I'll quickly know if it's a problem or a red herring. This works for any granularity of code, and is really valuable.</div></div></blockquote><blockquote type="cite"><div class="gmail_quote"><div><br>Without tests, each developer manually repeats the debugging via "Oh, I think it's caused when X happens and Y interacts with Z!". If you have a test for that that you commit to the repository, anyone who thinks it's broken can just re-run the test and see that no, that guess was wrong, try again.</div></div></blockquote><div><br></div><div>As you know Misha, LLVM has very strong interfaces. We have guarantees like "no pass should crash on a valid IR input". This is why things like bugpoint work. I personally have not suffered from the debugger issues that you are describing.</div><div><br></div><div>Most of the bugs that I have spent a lot of time scratching my head over are not logic bugs in the pass, they are things like dangling pointer and other fun bugs. The most insidious one is the "dangling pointer as the key of a map" which manifest non-determinstically when malloc happens to recycle a node, making a new node "already be in the map". This is a great reason to improve the LLVM infrastructure (e.g. the new AssertingVH<> class) to help make these bugs impossible. This would not have been caught with any reasonable testing. </div><div><br></div><blockquote type="cite"><div class="gmail_quote"><div>4) There are other things we cannot test via opt/grep directly, or even llvmirgrep:<br>(a) anything that doesn't have text-based output to grep through, e.g.<br> (i) analysis passes (alias analysis, etc.)</div></div></blockquote><div><br></div><div>These have text output with --aa-eval and friends. This is obviously true because we have many basicaa tests already.</div><br><blockquote type="cite"><div class="gmail_quote"><div> (ii) PassManager (e.g., verify pass ordering in the presence of FunctionPasses and ModulePasses, etc. that's independent of code being run)</div></div></blockquote><div><br></div><div>-debug-pass=Arguments / Structure still work :)</div><br><blockquote type="cite"><div class="gmail_quote"><div>(b) things that read or emit data are only tested indirectly: bitcode reader/writer,</div></div></blockquote><div><br></div><div>We have plenty of tests for these.</div><div><br></div><blockquote type="cite"><div class="gmail_quote"><div> TableGen, code emitters, etc.</div></div></blockquote><div><br></div>These get tested by whether or not the compiler works.</div><div><br><blockquote type="cite"><div class="gmail_quote"><div>Bottom line -- I see at least 2 open bugs on Andersen's that would benefit.</div></div></blockquote><div><br></div><div>Andersen's would benefit from contributors who care about it, not more tests :)</div><br><blockquote type="cite"><div class="gmail_quote"><div> How about we expose its class in a header to be able to write tests and see how it goes? And there is a bug for a pass that seems to run out of memory that might also benefit. And next time there's some difficult-to-debug problem that you see as similar to the above, please file a bug and assign it to me, I'll see what I can do for it with some targetted tests. How does that sound?</div></div></blockquote><div><br></div><div>No. Why does andersen's need any change to it to make it more testable? What part of its public interface is not testable?</div><br><blockquote type="cite"><div class="gmail_quote"><div><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> <div><div><div><blockquote type="cite"><div class="gmail_quote"><div><span>How else can we check the correctness of a pass, which involves pattern-matching multiple instructions? Grep can only look at a single line at a time, and doesn't have the semantic knowledge of what's an instruction and what's an operand, whereas we have all this information in C++.</span></div> </div> </blockquote></div></div><div>I think the answer is that we need to build *one* small "llvmirgrep" tool that does this, which could be used by many different tests.</div><div></div></div></blockquote><div> <br>This is a tool that doesn't exist yet, which means it will require time and effort to design and implement it, as well as (dare I say it?) test it. </div></div></blockquote><div><br></div><div>Of course. This is an infrastructure investment to make future testing easier. Having llvm-irgrep would make it much easier to write tests (increasing the odds that they will actually be written) than having to write C++ code. In case you haven't noticed, C++ is a very low-level language, and not one that is awesome at pattern matching ;-)</div><div><br></div><div>-Chris</div></div></body></html>