[llvm-dev] FileCheck: using numeric variable defined on same line with caveats

[James Henderson via llvm-dev]

I think I already gave my opinion on one of the previous patches, regarding
CHECK-NOT, which approximately came to the same conclusion as what you've
got here, so +1 from me. I also think the CHECK-DAG example is not one to
care about. It seems to me that there's no guarantee what CHECK-DAG:
[[LINE_AFTER_FOO:.*]] would match, as, if I followed it correctly,
CHECK-DAGs don't have any guarantee of order within a group, so it could
match either the next line after BEGIN, the line after  [[#VAR1:]]
[[#VAR1+1]] or indeed any line before END.

James

On Thu, 11 Jun 2020 at 12:29, Thomas Preud'homme via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> Hi,
>
> TL;DR: Is it ok to allow numeric variables used on same line as defined
> except for CHECK-NOT and with false negatives?
>
> FileCheck does not currently allow a numeric variable from being used on
> the same line they were defined. I have a tentative patch to add that
> support but it comes with caveats so before going through review I'd like
> to get consensus on whether those caveats are acceptable.
>
> == The problem ==
>
> The problem with matching variables defined on the same line is that the
> matching is done separately from checking the numeric relation, because
> numeric relation cannot be expressed in regex. That is, when matching
> [[#VAR:]] [[#VAR+1]] FileCheck is first matching the input against ([0-9]+)
> ([0-9]+) and then the value of the two captured integer are checked.
>
> This can lead to at times confusing or downward wrong outcomes. Consider
> the following input with the CHECK pattern mentioned above:
>
> 10 12 13
>
> The regex would match numbers 10 and 12 and fail the CHECK directive
> despite 12 and 13 verifying the +1 relation. This could happen as a result
> of a change in the input after a new commit has landed. In the case of a
> CHECK directive, it would make the test regress and a developer would need
> to tighten the pattern somehow, for instance by chaning it for [[#VAR:]]
> [[#VAR+1]]{{$}}. Now in the context of a CHECK-NOT this could be a change
> from input 10 12 14 to 10 12 13 and the pattern would still fail to match
> and thus the test still pass despite the compiler having regressed.
>
> == Proposed "solution" ==
>
> Given the above, we can summarize the risks of supporting numeric
> expression using a variable defined on the same line to:
>
>
>    - test regression on positive matching directives (CHECK, CHECK-NEXT,
>    ...)
>    - silent compiler regression on negative matching directives
>    (CHECK-NOT)
>
> I am therefore proposing to prevent using numeric variables defined on the
> same line for negative matching directives but allow it for positive
> matching directives with a note in the documentation to be careful to make
> the pattern as tight as possible.
>
> == CHECK-DAG case ==
>
> CHECK-DAG is interesting because despite it being a positive matching
> directive, there's a risk with CHECK-DAG in case a test rely on the way
> CHECK-DAG is implemented. Consider the following directives which rely on
> each directive being matched in order:
>
> CHECK: BEGIN
> CHECK-DAG: [[#VAR1:]] [[#VAR1+1]]
> CHECK-DAG: FOO
> CHECK-DAG: [[LINE_AFTER_FOO:.*]]
> CHECK: END
> CHECK-NOT: [[LINE_AFTER_FOO]] BAZ
>
> This could be written if the line checked by the first CHECK-DAG is
> guaranteed to always be either before FOO or after the line after FOO. Now
> consider the following input that verifies this invariant:
>
> BEGIN
> 10 12 13
> FOO 10 11
> FOOBAR
> END
> 10 12 13 FOOBAR BAZ
>
> The expectation from the test author relying on the CHECK-DAG behavior
> would be for LINE_AFTER_FOO to have the value FOOBAR once the CHECK-DAG
> block has matched. However due to the caveats mentioned above it would end
> up being set to "10 12 13"  and thus the CHECK-NOT would pass because "10
> 12 13" is not followed by "BAZ". That's far fetched though, I'm not
> convinced we should worry about this beyond documenting CHECK-DAG as being
> able to match in any order.
>
>
> Thoughts?
>
> Best regards,
>
> Thomas
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200615/47b9ebab/attachment.html>