[llvm-dev] FileCheck idiom difficulties

[David Blaikie via llvm-dev]

On Wed, Nov 6, 2019 at 11:12 AM Reid Kleckner via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> I think we should think about changing llvm-readobj to produce more
> filecheck friendly output.
>
> This was a problem we used to have with LLVM debug info metadata: it
> wasn't very structured, and it was printed in some arbitrary order. Then
> Duncan (I think) added the DI* classes, which made it easier to match
> something semantic, like `DILocalVariable.*name: "foo"`, and standardized
> on a topological output ordering, so you could start from the variable,
> then match the type, and then find the type metadata number later.
>
> If we printed `Section (.foo) {` for example, that would help some.
>
> It still doesn't help establish delimited regions for properties printed
> across multiple lines for readability, though... Should we add some kind of
> ad-hoc delimiter balancing to FileCheck? Something like:
>
> CHECK-SCOPE: Section {
> CHECK: Name: asdf
> CHECK: Name: asdf
> CHECK-ENDSCOPE: }
>

There's some overlap between ^ and this existing construct \/

CHECK: Section {
CHECK-DAG: Name: asdf
CHECK-DAG: Name: asdf
CHECK: }


>
> A SCOPE directive line would have to end in a known delimiter, '(', '{',
> or '['. The ENDSCOPE directive would only match lines with delimiters that
> balance with the opening delimiter. It would kind of work for filechecking
> JSON, for example.
>
> The scope idea here is pretty half-baked, but it's food for thought.
>
> On Wed, Nov 6, 2019 at 4:18 AM James Henderson via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>>
>> Hi all,
>>
>> Many of our lit tests use FileCheck and a tool like llvm-readobj to check
>> properties of a section header/symbol/etc. A typical (pseudoised for
>> brevity) output to match against might be something like the following:
>>
>> Symbols [
>>   Symbol {
>>     Name: foo
>>     Value: 0
>>     Type: Function
>>     Section: .foo (1)
>>   }
>>   Symbol {
>>     Name: bar
>>     Value: 1
>>     Type: Object
>>     Section: .foo (1)
>>   }
>> ]
>>
>> and your lit test might want to check the properties of the foo symbol
>> like so:
>>
>> # CHECK:      Name: foo
>> # CHECK-NEXT: Value: 0
>> # CHECK-NEXT: Type: Function
>> # CHECK-NEXT: Section: .foo (1)
>>
>> This is fine. But what if you only care about the section of a symbol,
>> and not the value or type etc? You could do the following:
>>
>> # CHECK: Name: foo
>> # CHECK: Section: .foo (1)
>>
>> Hopefully some of you will already notice the problem with this approach:
>> if foo was in, say, the .baz section, the test will spuriously pass,
>> because the Section line will match the Section line for .bar. One
>> alternative to this is to explicitly match each field in between, using
>> CHECK-NEXT:
>>
>> # CHECK:      Name: foo
>> # CHECK-NEXT: Value:
>> # CHECK-NEXT: Type:
>> # CHECK-NEXT: Section: .foo (1)
>>
>> This works, but somewhat hides what is really being tested by adding
>> extra noise to the checks. In reality, there are actually other fields too
>> that need to be listed, meaning the "interesting" parts of the test are
>> even more hidden.
>>
>> I recently started using yet another approach:
>>
>> # CHECK: Name: foo
>> # CHECK: Section:
>> # CHECK-SAME:     .foo (1)
>>
>> This works because the Section: matched will be the first one found, i.e.
>> the one belonging to foo, and then .foo will be looked for on the same
>> line. However, I noticed today that this pattern has its own problem,
>> namely that there could be something between the Section tag and .foo. In
>> other words, the above pattern would match "Section: .bar.foo". A couple of
>> solutions to this are:
>>
>> # CHECK: Section:
>> # CHECK-NOT: {{[:graph:]}}
>> # CHECK-SAME: .foo (1)
>>
>> # CHECK: Section:
>> # CHECK-SAME: {{^}} .foo (1)
>>
>> The first one ensures that there's no non-whitespace between the end of
>> "Section:" and the start of ".foo (1)". The second ensures that the start
>> of the CHECK-SAME match is the "start of line", and since the first half of
>> the line has already been consumed, it means " .foo (1)" must immediately
>> follow "Section:". However, the first is even less readable than the
>> current CHECK-SAME approach, whilst the second is somewhat confusing if you
>> don't realise that FileCheck effectively consumes the things it has matched
>> already, so that they effectively don't exist any more.
>>
>> Does anybody have any other suggestions/thoughts/comments? One idea I had
>> was for a new directive something like "CHECK-IMMEDIATE" which is
>> implicitly the same as the final approach I suggested above, but maybe
>> adding a new directive to achieve this isn't the right approach?
>>
>> James
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191106/8b2630b3/attachment.html>