[Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support

Fri Sep 16 16:52:02 PDT 2016

I have been thinking about something similar. Ildb-mi specifically I have
serious concerns about because the code itself is a bit of an abomination.
It has its own entire test suite, which also doesn't work very well.

Using a tool like lldb mi is very similar in spirit to how llvm already
uses lit though. There is a collection of very specific tools that exist
just for testing, and the output of those is run through lit)

Lldb-mi was developed independently, had no review, and was essentially a
large code drop. and it also was not developed with testing in mind. I
agree we could probably make it work, but I've been in that code a few
times and I have to say, I don't want to go back.

In any case, i think we agree in principle that from a high level, we could
get a lot of benefits from this style of test.

I'll have more specifics on my thoughts in a few weeks

On Fri, Sep 16, 2016 at 4:07 PM Jason Molenda <jmolenda at apple.com> wrote:

> I thought about this more overnight and I'm more convinced that lit and
> lldb-mi make a great pair.  lldb-mi is a programmatic text format that
> isn't subject to the whims of command-line UI design over the years; well
> tests written in terms of MI will be resilient and stable.  lldb-mi MUCH
> more closely matches a non-lldb developer's view of how a debugger works.
> The syntax is different from what they use, but it isn't hard to figure
> out.  I haven't touched MI in four or five years and it only took me a
> couple minutes to figure out how to do run to a breakpoint and print a
> variable:
>
> -file-exec-and-symbols "/tmp/a.out"
>
> -break-insert a.c:10
>
> =breakpoint-modified,bkpt={number="6",type="breakpoint",disp="keep",enabled="y",addr="0x0000000100000f70",func="main",file="a.c",fullname="/tmp/a.c",line="10",times="0",original-location="a.c:10"}
>
> -exec-run
>
> -thread-info
>
> -exec-next
>
> -var-create var1 * c
>
> ^done,name="var1",numchild="0",value="12",type="int",thread-id="1",has_more="0"
>
>
> (I omitted the responses from all the command except -break-insert and
> -var-create to make it easier to read, but go run lldb-mi and play with it
> yourself or read the gdb MI documentation on the web.  It's a really simple
> format to work in.)
>
>
> Several of the proposal have been special commands in the lldb command
> line driver that have non-changing output styles or the like.  Why
> re-invent something that already exists?  This makes a ton more sense on
> every front.  Yes, it means you can't copy and paste your lldb debug
> session into a test case --- but we've all agreed that that's not something
> we want anyone doing anyway.
>
> I'm in favor of lit + lldb-mi and think this would add a lot of value as
> an additional way for people to write test cases.
>
>
> J
>
>
>
> > On Sep 15, 2016, at 7:40 PM, Zachary Turner <zturner at google.com> wrote:
> >
> > One thing I wonder about. It seems like everyone is mostly on the same
> page about command line output .
> >
> > What about input? Would anyone have objections to a test which ran a few
> commands to get the debugger into a particular state before doing something
> to verify the output? Let's assume I'm waving my hands about this last step
> but that it doesn't involve scraping the output of a debugger command
> >
> > Maybe this will never even be an issue, but I just want to make sure
> everyone is on the same page and that the objections are:
> >
> > a) specific to command OUTPUT, not input (i.e. it's ok to have a test
> run "break set -n main")
> > b) specific to *non trivial* output (checking that "p 5" displays 5 is
> ok)
> > c) specific to the the output of the *user* command line interface, so
> that some hypothetical other command line interface (which again I'm being
> hand wavy about) would not be subject to the same objections.
> >
> >
> > On Thu, Sep 15, 2016 at 7:28 PM Jason Molenda <jmolenda at apple.com>
> wrote:
> >
> > > On Sep 15, 2016, at 8:02 AM, Zachary Turner <zturner at google.com>
> wrote:
> > >
> > >
> > > It sounds like your goal is also "tests have to use the SB API and no
> other API", which if so I think that's counterproductive.   More
> productive, IMO, would be being open to any alternative that addresses the
> concerns you have with command-line tests.  There are more than 2 ways to
> skin a cat, so to speak.
> >
> > Thinking about this a bit, another approach would be to do lit tests on
> top of lldb-mi.  MI is a structured format for the debugger and a UI to
> communicate back and forth with a simple text markup language (it would be
> JSON if it were being done today, but it was added to gdb eighteen years
> ago, so it's not).  The commands correspond to the commands a debugger user
> would think to use -- no need to understand the structure of how lldb is
> implemented, like with the SB API.  The format is a little unusual for a
> human to type, but back when we supported gdb at Apple we worked in MI all
> the time (it was used to communicate between Xcode, our IDE, and gdb) by
> hand when testing and it was fine. "-exec-run" instead of run, that kind of
> thing.  I think there are four dozens different commands.
> >
> > lldb-mi itself uses SB API.  And the concerns about hardcoding command
> line UI don't apply, it's a key-value format intended for computers, no one
> is going to add an extra space character to anything -- the most it changes
> is that new key-value pairs are added to responses.
> >
> >
> > I agree there are many acceptable ways to write lit tests that don't
> involve lldb command line scraping, and I'm all in favor of using lit with
> tests that don't do that.  Of course the patch we're discussing has lit
> test examples that contradict our own policy on writing tests.  Once lit is
> supported in lldb, are we going to reject any testsuite additions that
> depend on the text output from the command line lldb?  If we're all on the
> same page here, then I have no reservations.
> >
> > Just to say out loud the future I can easily see:  We add lit, then we
> have people helpfully write a few dozen tests in lit that depend on the
> command line debugger output.  Those patches have to be rejected.
> >
> > J
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-commits/attachments/20160916/89a2ac83/attachment.html>