[PATCH] Add support to FileCheck for out-of-order matching

[Hal Finkel]

At the risk of being off-topic: It would be really nice to be able to write regression tests that don't depend on the order chosen by the instruction scheduler. I think that, for that, it would be useful to be able to define groups of check lines for which the order does not matter. Maybe something like this:

; CHECK-GROUP-BEGIN
; CHECK: add [[REG1:r[0-9]+]], r1, r2
; CHECK: add [[REG2:r[0-9]+]], r3, r4
; CHECK: mul [[REG1]], [[REG2]]
; CHECK-GROUP-END

The first two add instructions are independent, and I don't care in which order they appear. The dependence of the mul on the adds is still captured by the named regex patterns. FileCheck could check all patterns in the group at each line, removing those that have matched from consideration.

What do you think?

Thanks again,
Hal

----- Original Message -----
> From: "Reid Kleckner" <rnk at google.com>
> To: "Eli Bendersky" <eliben at google.com>
> Cc: "llvm-commits" <llvm-commits at cs.uiuc.edu>
> Sent: Wednesday, April 10, 2013 4:20:40 PM
> Subject: Re: [PATCH] Add support to FileCheck for out-of-order matching
> 
> I was thinking more like this could succeed:
> ; CHECK: foo
> ; CHECK-ANYWHERE: bar
> ; CHECK-NEXT: baz
> 
> bar <- second
> baz <- third
> foo  <- first match
> 
> The goal being to handle something like the vtable layout tests,
> which
> would look like:
> 
> // RUN: %clang_cc1 %s -triple=x86_64-apple-darwin10 -emit-llvm-only
> -fdump-vtable-layouts |& FileCheck %s
> ...
> namespace Test1 {
> // CHECK-ANYWHERE:      Vtable for 'Test1::A' (3 entries).
> // CHECK-NEXT:   0 | offset_to_top (0)
> // CHECK-NEXT:   1 | Test1::A RTTI
> // CHECK-NEXT:       -- (Test1::A, 0) vtable address --
> // CHECK-NEXT:   2 | void Test1::A::f()
> struct A {
>   virtual void f();
> };
> void A::f() { }
> }
> 
> Instead of the current repeated FileChecks with prefixes:
> 
> // RUN: %clang_cc1 %s -triple=x86_64-apple-darwin10 -emit-llvm-only
> -fdump-vtable-layouts > %t 2>&1
> // RUN: FileCheck --check-prefix=CHECK-1 %s < %t
> // RUN: FileCheck --check-prefix=CHECK-2 %s < %t
> ...
> // CHECK-1:      Vtable for 'Test1::A' (3 entries).
> // CHECK-1-NEXT:   0 | offset_to_top (0)
> // CHECK-1-NEXT:   1 | Test1::A RTTI
> // CHECK-1-NEXT:       -- (Test1::A, 0) vtable address --
> // CHECK-1-NEXT:   2 | void Test1::A::f()
> 
> On Wed, Apr 10, 2013 at 5:01 PM, Eli Bendersky <eliben at google.com>
> wrote:
> >
> >
> >
> > On Wed, Apr 10, 2013 at 2:00 PM, Eli Bendersky <eliben at google.com>
> > wrote:
> >>
> >> On Wed, Apr 10, 2013 at 1:40 PM, Reid Kleckner <rnk at google.com>
> >> wrote:
> >>>
> >>> It seems like we have a bunch of tests where we really don't care
> >>> about ordering.  Should we add support to FileCheck for that?
> >>>
> >>> For example, this test uses repeated filecheck runs on the temp
> >>> file
> >>> output with different prefixes just to work around this problem:
> >>>
> >>> http://llvm.org/viewvc/llvm-project/cfe/trunk/test/CodeGenCXX/vtable-layout.cpp?view=markup
> >>> I had the same problem while working on vbtable layout.
> >>>
> >>> This is a big grep test that can't easily be filecheck-ified
> >>> because
> >>> there's no out of order matching:
> >>>
> >>> http://llvm.org/viewvc/llvm-project/cfe/trunk/test/CodeGenObjC/metadata-symbols-64.m?revision=174461&view=markup
> >>> I'd like to filecheck-ify that test because there's something
> >>> wrong
> >>> with the grep escaping on my system.
> >>>
> >>> I propose CHECK-ANYWHERE, or something else with the same
> >>> behavior.
> >>> Basically, this is a check directive that just resets the
> >>> matching
> >>> cursor back the beginning of the file.  Any subsequent checks
> >>> follow
> >>> from the end of a successful match.  Any captured variables are
> >>> preserved, mostly because removing them would be hard.
> >>
> >>
> >> The proposal is interesting, but I have doubts about the proposed
> >> implementation. Wouldn't this make the semantics of some tests
> >> very...
> >> unusual? Consider this:
> >>
> >> CHECK: foo
> >> CHECK-EVERYWHERE: bar
> >> CHECK: baz
> >>
> >> Is this going to match:
> >>
> >> baz
> >> foo
> >> bar
> >>
> >> That's kind-of weird :-)
> >
> >
> > I figured I should be more constructive :-)
> > An alternative proposal without this problem (AFAICS) is to collect
> > all
> > "-EVERYWHERE" checks and run them separately in the end on the
> > whole file.
> > They're order-independent, so this seems safe. And oh while we're
> > at it,
> > let's call it CHECK-GLOBAL and also have CHECK-GLOBAL-NOT because
> > that one
> > would be really useful for rewriting some 'not grep' tests.
> >
> > Eli
> >
> >
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>