[llvm-dev] FileCheck idiom difficulties

[James Henderson via llvm-dev]

Hi all,

Many of our lit tests use FileCheck and a tool like llvm-readobj to check
properties of a section header/symbol/etc. A typical (pseudoised for
brevity) output to match against might be something like the following:

Symbols [
  Symbol {
    Name: foo
    Value: 0
    Type: Function
    Section: .foo (1)
  }
  Symbol {
    Name: bar
    Value: 1
    Type: Object
    Section: .foo (1)
  }
]

and your lit test might want to check the properties of the foo symbol like
so:

# CHECK:      Name: foo
# CHECK-NEXT: Value: 0
# CHECK-NEXT: Type: Function
# CHECK-NEXT: Section: .foo (1)

This is fine. But what if you only care about the section of a symbol, and
not the value or type etc? You could do the following:

# CHECK: Name: foo
# CHECK: Section: .foo (1)

Hopefully some of you will already notice the problem with this approach:
if foo was in, say, the .baz section, the test will spuriously pass,
because the Section line will match the Section line for .bar. One
alternative to this is to explicitly match each field in between, using
CHECK-NEXT:

# CHECK:      Name: foo
# CHECK-NEXT: Value:
# CHECK-NEXT: Type:
# CHECK-NEXT: Section: .foo (1)

This works, but somewhat hides what is really being tested by adding extra
noise to the checks. In reality, there are actually other fields too that
need to be listed, meaning the "interesting" parts of the test are even
more hidden.

I recently started using yet another approach:

# CHECK: Name: foo
# CHECK: Section:
# CHECK-SAME:     .foo (1)

This works because the Section: matched will be the first one found, i.e.
the one belonging to foo, and then .foo will be looked for on the same
line. However, I noticed today that this pattern has its own problem,
namely that there could be something between the Section tag and .foo. In
other words, the above pattern would match "Section: .bar.foo". A couple of
solutions to this are:

# CHECK: Section:
# CHECK-NOT: {{[:graph:]}}
# CHECK-SAME: .foo (1)

# CHECK: Section:
# CHECK-SAME: {{^}} .foo (1)

The first one ensures that there's no non-whitespace between the end of
"Section:" and the start of ".foo (1)". The second ensures that the start
of the CHECK-SAME match is the "start of line", and since the first half of
the line has already been consumed, it means " .foo (1)" must immediately
follow "Section:". However, the first is even less readable than the
current CHECK-SAME approach, whilst the second is somewhat confusing if you
don't realise that FileCheck effectively consumes the things it has matched
already, so that they effectively don't exist any more.

Does anybody have any other suggestions/thoughts/comments? One idea I had
was for a new directive something like "CHECK-IMMEDIATE" which is
implicitly the same as the final approach I suggested above, but maybe
adding a new directive to achieve this isn't the right approach?

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191106/055f9807/attachment.html>