[Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support

[Jason Molenda via lldb-commits]

It's great to make writing tests easier.  We'd all love to have more tests.

If "writing tests easier" is "command line output scraping", that's only hurting the project in the long term.  I'm telling you this from years of experience on a project where we did exactly this.  It's great to have examples where we're testing that "p 5" prints 5.  That's awesome and I'm sure it won't break.  But that's not more than a tiny fraction of what you need to test in a debugger. 

It's important to separate "let's use lit" and "let's write command line scraping tests", unless lit means "command line scraping tests", in which case it's fine to conflate the two.  No one is going to be up in arms about lit as a test harness -- but using that as a way to get lots of command line scraping tests added to lldb is disingenuous.  We've had this discussion many times on the mailing lists, and all of us who had to work on a scraping testsuite have pushed back hard against the siren song of these tests because we've learned what a big mistake it is.

Talking about python 2 v python 3 compatibility or speed of the testsuite -- these are implementation side issues that I don't have an opinion on.  If we want to write our SB API in a different way to solve those issues -- want to write the tests in C++?  Fine, whatever.  I think that would just make it slower to write new tests.  Can tests in lit use the SB API exclusively?  Then I don't care about adding lit.

I remain firmly against command line scraping tests, it only adds debt to the project long term.  It's a lesson learned from gdb that bears not repeating.

J


> On Sep 14, 2016, at 9:36 PM, Zachary Turner <zturner at google.com> wrote:
> 
> There's also a cost to *not* writing test cases, which is paid proportionally to how difficult it is to write test cases.
> 
> That said, aren't the existing lldbinline tests an example of the behavior you're objecting to? And that is a relatively recent addition which I have never seen or heard of any objections to.
> 
> On Wed, Sep 14, 2016 at 9:07 PM Jason Molenda <jmolenda at apple.com> wrote:
> 
> On Sep 14, 2016, at 5:13 PM, Zachary Turner <zturner at google.com> wrote:
> 
> > I don't blame you for being scared of command tests.  I don't support their use in the current LLDB test suite either, for exactly the same reasons you and Jason have expressed.  But I do think it's possible to come up with something that a) doesn't suffer from the same problems, b) allows testing a ton of extra functionality that is not currently testable through the api, and c) doesn't rely on python at all.  If I'm wrong I'll eat crow :)
> 
> 
> I think your examples are reductive to "printing values is easy to keep consistent" -- but that's only a small part of what a debugger testsuite needs to do.
> 
> If we want to use lit to drive tests written in terms of SB API, then I'm open to seeing how this tool can be used in lldb.
> 
> If lit can only be used to write command output tests, then I believe, based on years and years of experience with exactly that kind of test suite in with gdb, that this is a very poor addition to lldb.  It will only hamper future lldb development as we want to improve the lldb command line UI - as it did with gdb.  Every change to the command line UI breaks hundreds of "easy to write" tests.  And who is responsible for cleaning all those up?  The people who wrote these easy-to-write command line output matching tests?  No, it'll be the person trying to improve the debugger, which is a big disincentive to changing anything else you'll need to wade through the swamp of thousands of accumulated test cases.
> 
> There's a cost to writing a test case.  Either it is paid up front when you write the test in terms of SB API, or it is paid repeatedly over time as people change the UI to the command line tool.
> 
> I have nothing against lit per se, I'm open to seeing what an SB API based test case in that harness looks like.  I have nothing but objections to adding significant new additions to the testsuite that are locked to the command line UI behavior of lldb today.
> 
> 
> J