[Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support

[Zachary Turner via lldb-commits]

On Wed, Sep 14, 2016 at 11:43 PM Jason Molenda <jmolenda at apple.com> wrote:

> If we want to add a testsuite runner which takes a source file, a place to
> put a breakpoint, the name of a variable to examine, and it runs through
> those in SB API, I'm all in favor.  I don't know if that's going to add a
> lot of test coverage to lldb, but I have no problem with such a thing.
>
> My goal isn't to make test case writing hard.  My goal is to make the
> testsuite a benefit to the project, instead of a boat anchor.
>
> My non-goal is what testsuite harness is used to run the SB API tests -
> whether it's lit or whatever, I don't care.
>

It sounds like your goal is also "tests have to use the SB API and no other
API", which if so I think that's counterproductive.   More productive, IMO,
would be being open to any alternative that addresses the concerns you have
with command-line tests.  There are more than 2 ways to skin a cat, so to
speak.

Aside from that, it sounds like we have the same goal -- make the test
suite a benefit to the project rather than a boat anchor.  I guess we
differ on what constitutes a benefit and a boat anchor.  TBH I think the
current test suite is a boat anchor due to its numerous stability issues
and the barrier to entry for writing new tests.  lit wins by leaps and
bounds in both of these areas.  If there were something else that were even
better, then I'd be all for using it (although then we'd have to question
why LLVM doesn't also use it).  But the fact is -- as a test
infrastructure, lit is better.



On Wed, Sep 14, 2016 at 11:19 PM Jason Molenda <jmolenda at apple.com> wrote:

> It's great to make writing tests easier.  We'd all love to have more tests.
>
> If "writing tests easier" is "command line output scraping", that's only
> hurting the project in the long term.  I'm telling you this from years of
> experience on a project where we did exactly this.  It's great to have
> examples where we're testing that "p 5" prints 5.  That's awesome and I'm
> sure it won't break.  But that's not more than a tiny fraction of what you
> need to test in a debugger.


At least 2 different brainstorm ideas have been proposed that would make
lit tests not rely on "command line output scraping".  A command line is
just a particular form of an API.  The SB API is another form of an API.
There are infinite numbers of ways to design an API, these aren't the only
2 APIs in the world.  One could easily imagine an "API" that was itself a
single LLDB command and which you could pass some kind of argument to it
that resolved to an SB API call.  Sean gave an example earlier.

(lldb) script lldb.frame.FindVariable("argc").GetValue()
'1'
(lldb) script lldb.process.GetNumThreads()
1
(lldb) script lldb.process.GetThreadAtIndex(0).GetThreadID()
3514809

Assuming your test ran against this output, is this a "command line
scraping" test or an "SB API" test?  Seems to be both, which is why I keep
saying that it's unproductive to talk about SB API tests and command line
scraping tests as the entire universe of options.

But even the above is not the only possibility.  You could directly map
every single SB API call to an LLDB command-line independent of the normal
command line UI.  One-to-one mapping

(lldb) object lldb.frame.variables["argc"].value
'1'
(lldb) object lldb.process.threads.count
1
(lldb) object lldb.process.threads[0].id
3514809

Is this really any different than the script example above?

So I would request that we stop talking about "command line scraping
tests".  Let's focus on the actual issue, which is "tests that are hard to
maintain: bad.  Tests that are easy to maintain: good"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-commits/attachments/20160915/abc80652/attachment.html>