<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
TL;DR: Is it ok to allow numeric variables used on same line as defined except for CHECK-NOT and with false negatives?<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
FileCheck does not currently allow a numeric variable from being used on the same line they were defined. I have a tentative patch to add that support but it comes with caveats so before going through review I'd like to get consensus on whether those caveats
are acceptable.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
== The problem ==</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The problem with matching variables defined on the same line is that the matching is done separately from checking the numeric relation, because numeric relation cannot be expressed in regex. That is, when matching [[#VAR:]] [[#VAR+1]] FileCheck is first matching
the input against ([0-9]+) ([0-9]+) and then the value of the two captured integer are checked.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
This can lead to at times confusing or downward wrong outcomes. Consider the following input with the CHECK pattern mentioned above:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
10 12 13</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The regex would match numbers 10 and 12 and fail the CHECK directive despite 12 and 13 verifying the +1 relation. This could happen as a result of a change in the input after a new commit has landed. In the case of a CHECK directive, it would make the test
regress and a developer would need to tighten the pattern somehow, for instance by chaning it for [[#VAR:]] [[#VAR+1]]{{$}}. Now in the context of a CHECK-NOT this could be a change from input 10 12 14 to 10 12 13 and the pattern would still fail to match
and thus the test still pass despite the compiler having regressed.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
== Proposed "solution" ==</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Given the above, we can summarize the risks of supporting numeric expression using a variable defined on the same line to:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<ul>
<li>test regression on positive matching directives (CHECK, CHECK-NEXT, ...)</li><li>silent compiler regression on negative matching directives (CHECK-NOT)<br>
</li></ul>
<div>I am therefore proposing to prevent using numeric variables defined on the same line for negative matching directives but allow it for positive matching directives with a note in the documentation to be careful to make the pattern as tight as possible.</div>
<div><br>
</div>
<div>== CHECK-DAG case ==<br>
</div>
<div><br>
</div>
<div>CHECK-DAG is interesting because despite it being a positive matching directive, there's a risk with CHECK-DAG in case a test rely on the way CHECK-DAG is implemented. Consider the following directives which rely on each directive being matched in order:<br>
</div>
<div><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace;">CHECK: BEGIN</span></div>
<div><span style="font-family: Consolas, Courier, monospace;">CHECK-DAG: [[#VAR1:]] [[#VAR1+1]]</span></div>
<div><span style="font-family: Consolas, Courier, monospace;">CHECK-DAG: FOO<br>
</span></div>
<div><span style="font-family: Consolas, Courier, monospace;">CHECK-DAG: [[LINE_AFTER_FOO:.*]]</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace;">CHECK: END</span></div>
<div><span style="font-family: Consolas, Courier, monospace;">CHECK-NOT: [[LINE_AFTER_FOO]] BAZ<br>
</span></div>
<div><span style="font-family: Consolas, Courier, monospace;"><br>
</span></div>
<div><span style="font-family: Calibri, Helvetica, sans-serif;">This could be written if the line checked by the first CHECK-DAG is guaranteed to always be either before FOO or after the line after FOO. Now consider the following input that verifies this invariant:</span></div>
<div><span style="font-family: Calibri, Helvetica, sans-serif;"><br>
</span></div>
<div><span style="font-family: Consolas, Courier, monospace;">BEGIN<br>
</span></div>
<div><span style="font-family: Consolas, Courier, monospace;">10 12 13</span></div>
<div><span style="font-family: Consolas, Courier, monospace;">FOO 10 11<br>
</span></div>
<div><span style="font-family: Consolas, Courier, monospace;">FOOBAR<br>
</span></div>
<div><span style="font-family: Consolas, Courier, monospace;">END<br>
</span></div>
<div><span style="font-family: Consolas, Courier, monospace;">10 12 13 FOOBAR BAZ<br>
</span></div>
<div><span style="font-family: Consolas, Courier, monospace;"><br>
</span></div>
<div>The expectation from the test author relying on the CHECK-DAG behavior would be for LINE_AFTER_FOO to have the value FOOBAR once the CHECK-DAG block has matched. However due to the caveats mentioned above it would end up being set to "10 12 13" and thus
the CHECK-NOT would pass because "10 12 13" is not followed by "BAZ". That's far fetched though, I'm not convinced we should worry about this beyond documenting CHECK-DAG as being able to match in any order.</div>
<div><br>
</div>
<div><br>
</div>
<div>Thoughts?</div>
<div><br>
</div>
<div>Best regards,</div>
<div><br>
</div>
<div>Thomas<br>
</div>
</div>
</body>
</html>