[Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support

[Jason Molenda via lldb-commits]

I thought about this more overnight and I'm more convinced that lit and lldb-mi make a great pair.  lldb-mi is a programmatic text format that isn't subject to the whims of command-line UI design over the years; well tests written in terms of MI will be resilient and stable.  lldb-mi MUCH more closely matches a non-lldb developer's view of how a debugger works.  The syntax is different from what they use, but it isn't hard to figure out.  I haven't touched MI in four or five years and it only took me a couple minutes to figure out how to do run to a breakpoint and print a variable:

-file-exec-and-symbols "/tmp/a.out"

-break-insert a.c:10
=breakpoint-modified,bkpt={number="6",type="breakpoint",disp="keep",enabled="y",addr="0x0000000100000f70",func="main",file="a.c",fullname="/tmp/a.c",line="10",times="0",original-location="a.c:10"}

-exec-run

-thread-info

-exec-next

-var-create var1 * c
^done,name="var1",numchild="0",value="12",type="int",thread-id="1",has_more="0"


(I omitted the responses from all the command except -break-insert and -var-create to make it easier to read, but go run lldb-mi and play with it yourself or read the gdb MI documentation on the web.  It's a really simple format to work in.)


Several of the proposal have been special commands in the lldb command line driver that have non-changing output styles or the like.  Why re-invent something that already exists?  This makes a ton more sense on every front.  Yes, it means you can't copy and paste your lldb debug session into a test case --- but we've all agreed that that's not something we want anyone doing anyway.

I'm in favor of lit + lldb-mi and think this would add a lot of value as an additional way for people to write test cases.


J



> On Sep 15, 2016, at 7:40 PM, Zachary Turner <zturner at google.com> wrote:
> 
> One thing I wonder about. It seems like everyone is mostly on the same page about command line output .
> 
> What about input? Would anyone have objections to a test which ran a few commands to get the debugger into a particular state before doing something to verify the output? Let's assume I'm waving my hands about this last step but that it doesn't involve scraping the output of a debugger command 
> 
> Maybe this will never even be an issue, but I just want to make sure everyone is on the same page and that the objections are:
> 
> a) specific to command OUTPUT, not input (i.e. it's ok to have a test run "break set -n main")
> b) specific to *non trivial* output (checking that "p 5" displays 5 is ok)
> c) specific to the the output of the *user* command line interface, so that some hypothetical other command line interface (which again I'm being hand wavy about) would not be subject to the same objections.
> 
> 
> On Thu, Sep 15, 2016 at 7:28 PM Jason Molenda <jmolenda at apple.com> wrote:
> 
> > On Sep 15, 2016, at 8:02 AM, Zachary Turner <zturner at google.com> wrote:
> >
> >
> > It sounds like your goal is also "tests have to use the SB API and no other API", which if so I think that's counterproductive.   More productive, IMO, would be being open to any alternative that addresses the concerns you have with command-line tests.  There are more than 2 ways to skin a cat, so to speak.
> 
> Thinking about this a bit, another approach would be to do lit tests on top of lldb-mi.  MI is a structured format for the debugger and a UI to communicate back and forth with a simple text markup language (it would be JSON if it were being done today, but it was added to gdb eighteen years ago, so it's not).  The commands correspond to the commands a debugger user would think to use -- no need to understand the structure of how lldb is implemented, like with the SB API.  The format is a little unusual for a human to type, but back when we supported gdb at Apple we worked in MI all the time (it was used to communicate between Xcode, our IDE, and gdb) by hand when testing and it was fine. "-exec-run" instead of run, that kind of thing.  I think there are four dozens different commands.
> 
> lldb-mi itself uses SB API.  And the concerns about hardcoding command line UI don't apply, it's a key-value format intended for computers, no one is going to add an extra space character to anything -- the most it changes is that new key-value pairs are added to responses.
> 
> 
> I agree there are many acceptable ways to write lit tests that don't involve lldb command line scraping, and I'm all in favor of using lit with tests that don't do that.  Of course the patch we're discussing has lit test examples that contradict our own policy on writing tests.  Once lit is supported in lldb, are we going to reject any testsuite additions that depend on the text output from the command line lldb?  If we're all on the same page here, then I have no reservations.
> 
> Just to say out loud the future I can easily see:  We add lit, then we have people helpfully write a few dozen tests in lit that depend on the command line debugger output.  Those patches have to be rejected.
> 
> J