[PATCH] Add support to FileCheck for out-of-order matching

Wed Apr 10 15:33:27 PDT 2013

On Wed, Apr 10, 2013 at 2:31 PM, Hal Finkel <hfinkel at anl.gov> wrote:

> At the risk of being off-topic: It would be really nice to be able to
> write regression tests that don't depend on the order chosen by the
> instruction scheduler. I think that, for that, it would be useful to be
> able to define groups of check lines for which the order does not matter.
> Maybe something like this:
>
> ; CHECK-GROUP-BEGIN
> ; CHECK: add [[REG1:r[0-9]+]], r1, r2
> ; CHECK: add [[REG2:r[0-9]+]], r3, r4
> ; CHECK: mul [[REG1]], [[REG2]]
> ; CHECK-GROUP-END
>
> The first two add instructions are independent, and I don't care in which
> order they appear. The dependence of the mul on the adds is still captured
> by the named regex patterns. FileCheck could check all patterns in the
> group at each line, removing those that have matched from consideration.
>
> What do you think?
>

This has been rehashed repeatedly on the lists. If you want to re-open
those discussions, cite the history and what new information is motivating
it.

For folks who missed the discussion, I (and others) have pushed back
against this because fundamentally the output of the compiler should be
deterministic and stable, even if arbitrary. So encoding a particular
arbitrary ordering, while annoying when updating, hasn't historically been
a huge burden. It might be a huge burden if it occurs in a very large
number of places causing very large churn on innocent changes, but so far
it has only come up in a small enough number of places to not be a huge
issue. (Register names on the other hand when hard coded did become such a
burden, and we've moved to consistently not hard code registers where doing
so doesn't make sense.)

I actually wonder whether the above rationale applies to Reid's situation
as well. I'd like to better understand why writing a particular output
creates a maintenance problem.

Regarding the "-NOT" version, I don't really see the need as the -NOT ones
are essentially order independent amongst themselves, and only ordered
w.r.t. the positive assertions...

>
> Thanks again,
> Hal
>
> ----- Original Message -----
> > From: "Reid Kleckner" <rnk at google.com>
> > To: "Eli Bendersky" <eliben at google.com>
> > Cc: "llvm-commits" <llvm-commits at cs.uiuc.edu>
> > Sent: Wednesday, April 10, 2013 4:20:40 PM
> > Subject: Re: [PATCH] Add support to FileCheck for out-of-order matching
> >
> > I was thinking more like this could succeed:
> > ; CHECK: foo
> > ; CHECK-ANYWHERE: bar
> > ; CHECK-NEXT: baz
> >
> > bar <- second
> > baz <- third
> > foo  <- first match
> >
> > The goal being to handle something like the vtable layout tests,
> > which
> > would look like:
> >
> > // RUN: %clang_cc1 %s -triple=x86_64-apple-darwin10 -emit-llvm-only
> > -fdump-vtable-layouts |& FileCheck %s
> > ...
> > namespace Test1 {
> > // CHECK-ANYWHERE:      Vtable for 'Test1::A' (3 entries).
> > // CHECK-NEXT:   0 | offset_to_top (0)
> > // CHECK-NEXT:   1 | Test1::A RTTI
> > // CHECK-NEXT:       -- (Test1::A, 0) vtable address --
> > // CHECK-NEXT:   2 | void Test1::A::f()
> > struct A {
> >   virtual void f();
> > };
> > void A::f() { }
> > }
> >
> > Instead of the current repeated FileChecks with prefixes:
> >
> > // RUN: %clang_cc1 %s -triple=x86_64-apple-darwin10 -emit-llvm-only
> > -fdump-vtable-layouts > %t 2>&1
> > // RUN: FileCheck --check-prefix=CHECK-1 %s < %t
> > // RUN: FileCheck --check-prefix=CHECK-2 %s < %t
> > ...
> > // CHECK-1:      Vtable for 'Test1::A' (3 entries).
> > // CHECK-1-NEXT:   0 | offset_to_top (0)
> > // CHECK-1-NEXT:   1 | Test1::A RTTI
> > // CHECK-1-NEXT:       -- (Test1::A, 0) vtable address --
> > // CHECK-1-NEXT:   2 | void Test1::A::f()
> >
> > On Wed, Apr 10, 2013 at 5:01 PM, Eli Bendersky <eliben at google.com>
> > wrote:
> > >
> > >
> > >
> > > On Wed, Apr 10, 2013 at 2:00 PM, Eli Bendersky <eliben at google.com>
> > > wrote:
> > >>
> > >> On Wed, Apr 10, 2013 at 1:40 PM, Reid Kleckner <rnk at google.com>
> > >> wrote:
> > >>>
> > >>> It seems like we have a bunch of tests where we really don't care
> > >>> about ordering.  Should we add support to FileCheck for that?
> > >>>
> > >>> For example, this test uses repeated filecheck runs on the temp
> > >>> file
> > >>> output with different prefixes just to work around this problem:
> > >>>
> > >>>
> http://llvm.org/viewvc/llvm-project/cfe/trunk/test/CodeGenCXX/vtable-layout.cpp?view=markup
> > >>> I had the same problem while working on vbtable layout.
> > >>>
> > >>> This is a big grep test that can't easily be filecheck-ified
> > >>> because
> > >>> there's no out of order matching:
> > >>>
> > >>>
> http://llvm.org/viewvc/llvm-project/cfe/trunk/test/CodeGenObjC/metadata-symbols-64.m?revision=174461&view=markup
> > >>> I'd like to filecheck-ify that test because there's something
> > >>> wrong
> > >>> with the grep escaping on my system.
> > >>>
> > >>> I propose CHECK-ANYWHERE, or something else with the same
> > >>> behavior.
> > >>> Basically, this is a check directive that just resets the
> > >>> matching
> > >>> cursor back the beginning of the file.  Any subsequent checks
> > >>> follow
> > >>> from the end of a successful match.  Any captured variables are
> > >>> preserved, mostly because removing them would be hard.
> > >>
> > >>
> > >> The proposal is interesting, but I have doubts about the proposed
> > >> implementation. Wouldn't this make the semantics of some tests
> > >> very...
> > >> unusual? Consider this:
> > >>
> > >> CHECK: foo
> > >> CHECK-EVERYWHERE: bar
> > >> CHECK: baz
> > >>
> > >> Is this going to match:
> > >>
> > >> baz
> > >> foo
> > >> bar
> > >>
> > >> That's kind-of weird :-)
> > >
> > >
> > > I figured I should be more constructive :-)
> > > An alternative proposal without this problem (AFAICS) is to collect
> > > all
> > > "-EVERYWHERE" checks and run them separately in the end on the
> > > whole file.
> > > They're order-independent, so this seems safe. And oh while we're
> > > at it,
> > > let's call it CHECK-GLOBAL and also have CHECK-GLOBAL-NOT because
> > > that one
> > > would be really useful for rewriting some 'not grep' tests.
> > >
> > > Eli
> > >
> > >
> > _______________________________________________
> > llvm-commits mailing list
> > llvm-commits at cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
> >
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20130410/07f71ab5/attachment.html>