[lldb-dev] Using FileCheck in lldb inline tests

[Zachary Turner via lldb-dev]

What do your patches do, out of curiosity?

On Wed, Aug 15, 2018 at 12:45 PM Vedant Kumar <vsk at apple.com> wrote:

>
> On Aug 15, 2018, at 12:27 PM, Zachary Turner <zturner at google.com> wrote:
>
> Back to the original proposal, my biggest concern is that a single inline
> test could generate many FileCheck invocations.  This could cause
> measurable performance impact on the test suite.  Have you considered this?
>
>
> That's a good point. I hadn't considered that. My thoughts on that are;
>
> - It's relatively cheap to create a FileCheck process. If the build is
> (A|T)sanified, we can copy in a non-sanitized FileCheck to speed things up.
>
> - Based on the time it takes to run check-{llvm,clang} locally, which have
> ~56,000 FileCheck invocations, my intuition is that the overhead ought to
> be manageable.
>
> - The status quo is doing Python's re.search over a chunk of command
> output. My (unverified) intuition is that FileCheck won't be slower than
> that. Actually, FileCheck has an algorithmic advantage because it doesn't
> re-scan the input text from the beginning of the text each time it tries to
> match a substring. `self.expect` does.
>
>
>
> Another possible solution is what i mentioned earlier, basically to expose
> a debugger object model.  This would allow you to accomplish what you want
> without FileCheck, while simultaneously being making many other types of
> tests easier to write at the same time.  On the other hand, it’s a larger
> effort to create this system, but I think long term it would pay back
> enormously (it’s even useful as a general purpose debugger feature, not
> limited to testing)
>
>
> I'd volunteer to work on that. At the moment I really need to get some
> form of testing put together for my patches soon.
>
> vedant
>
>
> On Tue, Aug 14, 2018 at 5:31 PM Vedant Kumar via lldb-dev <
> lldb-dev at lists.llvm.org> wrote:
>
>> Hello,
>>
>> I'd like to make FileCheck available within lldb inline tests, in
>> addition to existing helpers like 'runCmd' and 'expect'.
>>
>> My motivation is that several tests I'm working on can't be made as
>> rigorous as they need to be without FileCheck-style checks. In particular,
>> the 'matching', 'substrs', and 'patterns' arguments to runCmd/expect don't
>> allow me to verify the ordering of checked input, to be stringent about
>> line numbers, or to capture & reuse snippets of text from the input stream.
>>
>> I'd curious to know if anyone else is interested or would be willing to
>> review this (https://reviews.llvm.org/D50751).
>>
>> Here's an example of an inline test which benefits from FileCheck-style
>> checking. This test is trying to check that certain frames appear in a
>> backtrace when stopped inside of the "sink" function. Notice that without
>> FileCheck, it's not possible to verify the order in which frames are
>> printed, and that dealing with line numbers would be cumbersome.
>>
>> ```
>> ---
>> a/lldb/packages/Python/lldbsuite/test/functionalities/tail_call_frames/unambiguous_sequence/main.cpp
>> +++
>> b/lldb/packages/Python/lldbsuite/test/functionalities/tail_call_frames/unambiguous_sequence/main.cpp
>> @@ -9,16 +9,21 @@
>>
>>  volatile int x;
>>
>> +// CHECK: frame #0: {{.*}}sink() at main.cpp:[[@LINE+2]] [opt]
>>  void __attribute__((noinline)) sink() {
>> -  x++; //% self.expect("bt", substrs = ['main', 'func1', 'func2',
>> 'func3', 'sink'])
>> +  x++; //% self.filecheck("bt", "main.cpp")
>>  }
>>
>> +// CHECK-NEXT: frame #1: {{.*}}func3() {{.*}}[opt] [artificial]
>>  void __attribute__((noinline)) func3() { sink(); /* tail */ }
>>
>> +// CHECK-NEXT: frame #2: {{.*}}func2() at main.cpp:[[@LINE+1]] [opt]
>>  void __attribute__((disable_tail_calls, noinline)) func2() { func3(); /*
>> regular */ }
>>
>> +// CHECK-NEXT: frame #3: {{.*}}func1() {{.*}}[opt] [artificial]
>>  void __attribute__((noinline)) func1() { func2(); /* tail */ }
>>
>> +// CHECK-NEXT: frame #4: {{.*}}main at main.cpp:[[@LINE+2]] [opt]
>>  int __attribute__((disable_tail_calls)) main() {
>>    func1(); /* regular */
>>    return 0;
>> ```
>>
>> For reference, here's the output of the "bt" command:
>>
>> ```
>> runCmd: bt
>> output: * thread #1, queue = 'com.apple.main-thread', stop reason =
>> breakpoint 1.1
>>   * frame #0: 0x000000010c6a6f64 a.out`sink() at main.cpp:14 [opt]
>>     frame #1: 0x000000010c6a6f70 a.out`func3() at main.cpp:15 [opt]
>> [artificial]
>>     frame #2: 0x000000010c6a6f89 a.out`func2() at main.cpp:21 [opt]
>>     frame #3: 0x000000010c6a6f90 a.out`func1() at main.cpp:21 [opt]
>> [artificial]
>>     frame #4: 0x000000010c6a6fa9 a.out`main at main.cpp:28 [opt]
>> ```
>>
>> thanks,
>> vedant
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20180815/230039cc/attachment-0001.html>