[lldb-dev] Using FileCheck in lldb inline tests

[Vedant Kumar via lldb-dev]

> On Aug 15, 2018, at 10:15 AM, Jim Ingham via lldb-dev <lldb-dev at lists.llvm.org> wrote:
> 
> 
> 
>> On Aug 14, 2018, at 7:48 PM, Zachary Turner via lldb-dev <lldb-dev at lists.llvm.org> wrote:
>> 
>> 
>> On Tue, Aug 14, 2018 at 6:58 PM Jason Molenda <jmolenda at apple.com> wrote:
>> 
>> 
>>> On Aug 14, 2018, at 6:39 PM, Zachary Turner <zturner at google.com> wrote:
>>> 
>>> Having bugs also makes the debugger harder to innovate in the future because it’s, not having tests leads to having bugs, and sb api tests leads to not having te
>> 
>> Yes, lldb does not have these problems -- because we learned from our decades working on gdb, and did not repeat that mistake.  To be honest, lldb is such a young debugger - barely a decade old, depending on how you count it, that ANY testsuite approach would be fine at this point.  Add a couple more decades and we'd be back into the hole that gdb was in.  {I have not worked on gdb in over a decade, so I don't know how their testing methodology may be today}
>> That doesn’t mean that the current approach is the final word.  As new people come onto the project, new ideas come forth and we should entertain them rather than deciding that all decisions are set in stone forever.
>> 
>> For example, the object model based approach I mentioned earlier would not have any of the problems that you’ve described from gdb.  Just because one set of problems has been solved doesn’t mean we should declare victory and say there’s no point in trying to solve the remaining problems too.  And right now, the problem is that we need to be coming up with a way to make tests easier to write so that people will actually write them
> 
> Just historically it is not true that we didn't have problems with command output scraping vrs. the testsuite.  The major offender in this regard was breakpoint setting/checking.  But the first time I had to go adjust a whole bunch of tests when I wanted to change the breakpoint output, I added test routines to do all this business, which mitigated the need for these changes.  It is true I haven't had to fiddle with this much then - though that is mostly because I was careful in how I checked output, and didn't check inessential aspects.  That's easy to do when you sit down and do it once, but if everybody just wings some regex every time they do a check you will end up with lots over-eager checks that cause problems down the road.

I definitely sympathize with this, but don't think the problems that arise in some tests which pattern-match against input are characteristic of all such tests. The lesson you're describing -- the need to be careful to only examine the interesting parts of the API/driver output -- applies generally.


> I also want to reiterate another point Jason made, which is that LLDB not only provides a command line interface, but it also provides an API.  Again, we've gone over this before but I'm pretty sure that at this point it's still true that in terms of number of users, far more people interact with lldb through the API (Xcode, VSCode through the gdb-mi or through Greg's new adaptor when that's done...) than interact with it through the command line.  And writing SB API tests when you add a new feature is a great way to ensure that you have thought about exposing your new feature to the SB API's correctly.  In that example, a GUI needs to have a way to indicate that these "tail call" frames are special and you shouldn't be worried that they have no local variables, even though we show you source for them that obviously has variables...

I agree with all of this. However, I don't take it to mean that there ought not be tests for the driver output.


> So we need ways to encourage lldb developers to be aware that the SB API's need to have (tested) access to new features in the debugger and not just the command line.  SB API tests are good way to encourage this.  We can also do it as part of reviews and the like, but I think it better done as part of your development than after the fact in reviews...  Again, that doesn't argue for their being the ONLY way to test things in lldb, but it does need to be part of an lldb developer's practice.

Totally agree.

vedant

> I like your idea of some kind of object model based output for test checking.  I somehow had the impression that that was your intent for lldb-test once it was a real boy, but maybe I was just projecting...  Relying on ad hoc dumping routines which are mostly for debugging purposes seemed okay as a boot-strap method, but not a great design long-term.  Teaching either the SB or the lldb_private objects to describe themselves in a structured way and using that to do testing seems great.
> 
> Jim
> 
> 
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> 
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev