[Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support

[Zachary Turner via lldb-commits]

Also, sb api and command line test are neither mutually exclusive nor the
only 2 options.

Sean mentioned the idea of a dsl for example. I mentioned the idea of a
command that is independent of the normal ui command line.

There are lots of possibilities if you look.

Also our current situation is far from perfect. There is flakiness on every
platform. Python is slow. There's complexity in supporting both Python 2
and 3. The test suite is about 80% smaller than it should be since so much
goes into writing one. I don't know if there's any buildbots at all running
tests due to how fragile everything is.

Using lit solves many of those problems immediately, and makes it possible
to solve the rest.

Maybe we've solved the problem of hard to maintain tests, which again I
never experienced, but it's hard to argue that we're in a good state.
On Wed, Sep 14, 2016 at 9:36 PM Zachary Turner <zturner at google.com> wrote:

> There's also a cost to *not* writing test cases, which is paid
> proportionally to how difficult it is to write test cases.
>
> That said, aren't the existing lldbinline tests an example of the behavior
> you're objecting to? And that is a relatively recent addition which I have
> never seen or heard of any objections to.
>
> On Wed, Sep 14, 2016 at 9:07 PM Jason Molenda <jmolenda at apple.com> wrote:
>
>
> On Sep 14, 2016, at 5:13 PM, Zachary Turner <zturner at google.com> wrote:
>
> > I don't blame you for being scared of command tests.  I don't support
> their use in the current LLDB test suite either, for exactly the same
> reasons you and Jason have expressed.  But I do think it's possible to come
> up with something that a) doesn't suffer from the same problems, b) allows
> testing a ton of extra functionality that is not currently testable through
> the api, and c) doesn't rely on python at all.  If I'm wrong I'll eat crow
> :)
>
>
> I think your examples are reductive to "printing values is easy to keep
> consistent" -- but that's only a small part of what a debugger testsuite
> needs to do.
>
> If we want to use lit to drive tests written in terms of SB API, then I'm
> open to seeing how this tool can be used in lldb.
>
> If lit can only be used to write command output tests, then I believe,
> based on years and years of experience with exactly that kind of test suite
> in with gdb, that this is a very poor addition to lldb.  It will only
> hamper future lldb development as we want to improve the lldb command line
> UI - as it did with gdb.  Every change to the command line UI breaks
> hundreds of "easy to write" tests.  And who is responsible for cleaning all
> those up?  The people who wrote these easy-to-write command line output
> matching tests?  No, it'll be the person trying to improve the debugger,
> which is a big disincentive to changing anything else you'll need to wade
> through the swamp of thousands of accumulated test cases.
>
> There's a cost to writing a test case.  Either it is paid up front when
> you write the test in terms of SB API, or it is paid repeatedly over time
> as people change the UI to the command line tool.
>
> I have nothing against lit per se, I'm open to seeing what an SB API based
> test case in that harness looks like.  I have nothing but objections to
> adding significant new additions to the testsuite that are locked to the
> command line UI behavior of lldb today.
>
>
> J
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-commits/attachments/20160915/cd9e3ac9/attachment.html>