[PATCH] Add support to FileCheck for out-of-order matching

[Reid Kleckner]

On Wed, Apr 10, 2013 at 3:33 PM, Chandler Carruth <chandlerc at google.com> wrote:
> On Wed, Apr 10, 2013 at 2:31 PM, Hal Finkel <hfinkel at anl.gov> wrote:
>>
>> At the risk of being off-topic: It would be really nice to be able to
>> write regression tests that don't depend on the order chosen by the
>> instruction scheduler. I think that, for that, it would be useful to be able
>> to define groups of check lines for which the order does not matter. Maybe
>> something like this:
>>
>> ; CHECK-GROUP-BEGIN
>> ; CHECK: add [[REG1:r[0-9]+]], r1, r2
>> ; CHECK: add [[REG2:r[0-9]+]], r3, r4
>> ; CHECK: mul [[REG1]], [[REG2]]
>> ; CHECK-GROUP-END
>>
>> The first two add instructions are independent, and I don't care in which
>> order they appear. The dependence of the mul on the adds is still captured
>> by the named regex patterns. FileCheck could check all patterns in the group
>> at each line, removing those that have matched from consideration.
>>
>> What do you think?

This would create a burden for test authors.  They would have to
enumerate all of the reasonably plausible instruction orderings of the
generated code, which seems hard to do correctly.  If this feature
were added but no one used it, you'd still have to go update a bunch
of tests if the scheduling changes.

> This has been rehashed repeatedly on the lists. If you want to re-open those
> discussions, cite the history and what new information is motivating it.
>
> For folks who missed the discussion, I (and others) have pushed back against
> this because fundamentally the output of the compiler should be
> deterministic and stable, even if arbitrary. So encoding a particular
> arbitrary ordering, while annoying when updating, hasn't historically been a
> huge burden. It might be a huge burden if it occurs in a very large number
> of places causing very large churn on innocent changes, but so far it has
> only come up in a small enough number of places to not be a huge issue.
> (Register names on the other hand when hard coded did become such a burden,
> and we've moved to consistently not hard code registers where doing so
> doesn't make sense.)
>
>
> I actually wonder whether the above rationale applies to Reid's situation as
> well. I'd like to better understand why writing a particular output creates
> a maintenance problem.

I really only have two use cases:
1. Converting grep tests that would need to be reordered.
2. Checking output of deferred constructs (vtables and other class data).

No one should be adding grep tests, so use case #1 should disappear.

I think we could fix #2 by ensuring that deferred constructs get
emitted in roughly the order the appear in the source, but that's much
more work than adding CHECK-ANYWHERE was.  :)

I also did a quick scan of the clang tests to find instances of '> %t'
followed by more than one 'FileCheck < %t', since that's the pattern
for working around ordering issues, and I only found 48 tests, some of
which were false positives.  IMO that's a very small fraction of the
total number of tests, so it doesn't seem worth adding this feature
for them.