[lldb-dev] Using FileCheck in lldb inline tests

[Vedant Kumar via lldb-dev]

It'd be easy to update FileCheck tests when changing the debugger (this happens all the time in clang/swift). OTOH, the verbosity of the python API means that fewer tests get written. I see a real need to make expressive tests easier to write.

vedant

> On Aug 14, 2018, at 5:38 PM, Jason Molenda <jmolenda at apple.com> wrote:
> 
> I'd argue against this approach because it's exactly why the lit tests don't run against the lldb driver -- they're hardcoding the output of the lldb driver command into the testsuite and these will eventually make it much more difficult to change and improve the driver as we've accumulated this style of test.
> 
> This is a perfect test for a normal SB API.  Run to your breakpoints and check the stack frames.
> 
>  f0 = thread.GetFrameAtIndex(0)
>  check that f0.GetFunctionName() == sink
>  check that f0.IsArtifical() == True
>  check that f0.GetLineEntry().GetLine() == expected line number
> 
> 
> it's more verbose, but it's also much more explicit about what it's checking, and easy to see what has changed if there is a failure.
> 
> 
> J
> 
>> On Aug 14, 2018, at 5:31 PM, Vedant Kumar via lldb-dev <lldb-dev at lists.llvm.org> wrote:
>> 
>> Hello,
>> 
>> I'd like to make FileCheck available within lldb inline tests, in addition to existing helpers like 'runCmd' and 'expect'.
>> 
>> My motivation is that several tests I'm working on can't be made as rigorous as they need to be without FileCheck-style checks. In particular, the 'matching', 'substrs', and 'patterns' arguments to runCmd/expect don't allow me to verify the ordering of checked input, to be stringent about line numbers, or to capture & reuse snippets of text from the input stream.
>> 
>> I'd curious to know if anyone else is interested or would be willing to review this (https://reviews.llvm.org/D50751).
>> 
>> Here's an example of an inline test which benefits from FileCheck-style checking. This test is trying to check that certain frames appear in a backtrace when stopped inside of the "sink" function. Notice that without FileCheck, it's not possible to verify the order in which frames are printed, and that dealing with line numbers would be cumbersome.
>> 
>> ```
>> --- a/lldb/packages/Python/lldbsuite/test/functionalities/tail_call_frames/unambiguous_sequence/main.cpp
>> +++ b/lldb/packages/Python/lldbsuite/test/functionalities/tail_call_frames/unambiguous_sequence/main.cpp
>> @@ -9,16 +9,21 @@
>> 
>> volatile int x;
>> 
>> +// CHECK: frame #0: {{.*}}sink() at main.cpp:[[@LINE+2]] [opt]
>> void __attribute__((noinline)) sink() {
>> -  x++; //% self.expect("bt", substrs = ['main', 'func1', 'func2', 'func3', 'sink'])
>> +  x++; //% self.filecheck("bt", "main.cpp")
>> }
>> 
>> +// CHECK-NEXT: frame #1: {{.*}}func3() {{.*}}[opt] [artificial]
>> void __attribute__((noinline)) func3() { sink(); /* tail */ }
>> 
>> +// CHECK-NEXT: frame #2: {{.*}}func2() at main.cpp:[[@LINE+1]] [opt]
>> void __attribute__((disable_tail_calls, noinline)) func2() { func3(); /* regular */ }
>> 
>> +// CHECK-NEXT: frame #3: {{.*}}func1() {{.*}}[opt] [artificial]
>> void __attribute__((noinline)) func1() { func2(); /* tail */ }
>> 
>> +// CHECK-NEXT: frame #4: {{.*}}main at main.cpp:[[@LINE+2]] [opt]
>> int __attribute__((disable_tail_calls)) main() {
>>   func1(); /* regular */
>>   return 0;
>> ```
>> 
>> For reference, here's the output of the "bt" command:
>> 
>> ```
>> runCmd: bt
>> output: * thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
>>  * frame #0: 0x000000010c6a6f64 a.out`sink() at main.cpp:14 [opt]
>>    frame #1: 0x000000010c6a6f70 a.out`func3() at main.cpp:15 [opt] [artificial]
>>    frame #2: 0x000000010c6a6f89 a.out`func2() at main.cpp:21 [opt]
>>    frame #3: 0x000000010c6a6f90 a.out`func1() at main.cpp:21 [opt] [artificial]
>>    frame #4: 0x000000010c6a6fa9 a.out`main at main.cpp:28 [opt]
>> ```
>> 
>> thanks,
>> vedant
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>