[PATCH] Add support to FileCheck for out-of-order matching
Nick Lewycky
nicholas at mxc.ca
Fri May 3 07:55:47 PDT 2013
Chandler Carruth wrote:
> On Wed, Apr 10, 2013 at 2:31 PM, Hal Finkel <hfinkel at anl.gov
> <mailto:hfinkel at anl.gov>> wrote:
>
> At the risk of being off-topic: It would be really nice to be able
> to write regression tests that don't depend on the order chosen by
> the instruction scheduler. I think that, for that, it would be
> useful to be able to define groups of check lines for which the
> order does not matter. Maybe something like this:
>
> ; CHECK-GROUP-BEGIN
> ; CHECK: add [[REG1:r[0-9]+]], r1, r2
> ; CHECK: add [[REG2:r[0-9]+]], r3, r4
> ; CHECK: mul [[REG1]], [[REG2]]
> ; CHECK-GROUP-END
>
> The first two add instructions are independent, and I don't care in
> which order they appear. The dependence of the mul on the adds is
> still captured by the named regex patterns. FileCheck could check
> all patterns in the group at each line, removing those that have
> matched from consideration.
>
> What do you think?
>
>
> This has been rehashed repeatedly on the lists. If you want to re-open
> those discussions, cite the history and what new information is
> motivating it.
>
> For folks who missed the discussion, I (and others) have pushed back
> against this because fundamentally the output of the compiler should be
> deterministic and stable, even if arbitrary. So encoding a particular
> arbitrary ordering, while annoying when updating, hasn't historically
> been a huge burden. It might be a huge burden if it occurs in a very
> large number of places causing very large churn on innocent changes, but
> so far it has only come up in a small enough number of places to not be
> a huge issue. (Register names on the other hand when hard coded did
> become such a burden, and we've moved to consistently not hard code
> registers where doing so doesn't make sense.)
One of the problems I have is that emitting a .ll file will reorder
things. So I group my function declarations, definitions and named
metadata together in a single logical test, then when emitting they're
spread all over the file (named metadata is grouped at the end). That's
really unfortunate, it happens a lot with debug info tests. This would
give me a way to group together my check statements with the logical test.
As an aside, another way to achieve this is with scoped/parenthesized
check statements. That is, I say that for my test1, X1 is before Y1 is
before Z1, but that makes no claims about the order of checks for test2,
where X2 is before Y2 is before Z2. The file may be ordered as X1 X2 Y1
Y2 Z2 Z1, and that would pass.
Nick
>
>
> I actually wonder whether the above rationale applies to Reid's
> situation as well. I'd like to better understand why writing a
> particular output creates a maintenance problem.
>
>
> Regarding the "-NOT" version, I don't really see the need as the -NOT
> ones are essentially order independent amongst themselves, and only
> ordered w.r.t. the positive assertions...
>
>
> Thanks again,
> Hal
>
> ----- Original Message -----
> > From: "Reid Kleckner" <rnk at google.com <mailto:rnk at google.com>>
> > To: "Eli Bendersky" <eliben at google.com <mailto:eliben at google.com>>
> > Cc: "llvm-commits" <llvm-commits at cs.uiuc.edu
> <mailto:llvm-commits at cs.uiuc.edu>>
> > Sent: Wednesday, April 10, 2013 4:20:40 PM
> > Subject: Re: [PATCH] Add support to FileCheck for out-of-order
> matching
> >
> > I was thinking more like this could succeed:
> > ; CHECK: foo
> > ; CHECK-ANYWHERE: bar
> > ; CHECK-NEXT: baz
> >
> > bar <- second
> > baz <- third
> > foo <- first match
> >
> > The goal being to handle something like the vtable layout tests,
> > which
> > would look like:
> >
> > // RUN: %clang_cc1 %s -triple=x86_64-apple-darwin10 -emit-llvm-only
> > -fdump-vtable-layouts |& FileCheck %s
> > ...
> > namespace Test1 {
> > // CHECK-ANYWHERE: Vtable for 'Test1::A' (3 entries).
> > // CHECK-NEXT: 0 | offset_to_top (0)
> > // CHECK-NEXT: 1 | Test1::A RTTI
> > // CHECK-NEXT: -- (Test1::A, 0) vtable address --
> > // CHECK-NEXT: 2 | void Test1::A::f()
> > struct A {
> > virtual void f();
> > };
> > void A::f() { }
> > }
> >
> > Instead of the current repeated FileChecks with prefixes:
> >
> > // RUN: %clang_cc1 %s -triple=x86_64-apple-darwin10 -emit-llvm-only
> > -fdump-vtable-layouts > %t 2>&1
> > // RUN: FileCheck --check-prefix=CHECK-1 %s < %t
> > // RUN: FileCheck --check-prefix=CHECK-2 %s < %t
> > ...
> > // CHECK-1: Vtable for 'Test1::A' (3 entries).
> > // CHECK-1-NEXT: 0 | offset_to_top (0)
> > // CHECK-1-NEXT: 1 | Test1::A RTTI
> > // CHECK-1-NEXT: -- (Test1::A, 0) vtable address --
> > // CHECK-1-NEXT: 2 | void Test1::A::f()
> >
> > On Wed, Apr 10, 2013 at 5:01 PM, Eli Bendersky <eliben at google.com
> <mailto:eliben at google.com>>
> > wrote:
> > >
> > >
> > >
> > > On Wed, Apr 10, 2013 at 2:00 PM, Eli Bendersky
> <eliben at google.com <mailto:eliben at google.com>>
> > > wrote:
> > >>
> > >> On Wed, Apr 10, 2013 at 1:40 PM, Reid Kleckner <rnk at google.com
> <mailto:rnk at google.com>>
> > >> wrote:
> > >>>
> > >>> It seems like we have a bunch of tests where we really don't care
> > >>> about ordering. Should we add support to FileCheck for that?
> > >>>
> > >>> For example, this test uses repeated filecheck runs on the temp
> > >>> file
> > >>> output with different prefixes just to work around this problem:
> > >>>
> > >>>
> http://llvm.org/viewvc/llvm-project/cfe/trunk/test/CodeGenCXX/vtable-layout.cpp?view=markup
> > >>> I had the same problem while working on vbtable layout.
> > >>>
> > >>> This is a big grep test that can't easily be filecheck-ified
> > >>> because
> > >>> there's no out of order matching:
> > >>>
> > >>>
> http://llvm.org/viewvc/llvm-project/cfe/trunk/test/CodeGenObjC/metadata-symbols-64.m?revision=174461&view=markup
> <http://llvm.org/viewvc/llvm-project/cfe/trunk/test/CodeGenObjC/metadata-symbols-64.m?revision=174461&view=markup>
> > >>> I'd like to filecheck-ify that test because there's something
> > >>> wrong
> > >>> with the grep escaping on my system.
> > >>>
> > >>> I propose CHECK-ANYWHERE, or something else with the same
> > >>> behavior.
> > >>> Basically, this is a check directive that just resets the
> > >>> matching
> > >>> cursor back the beginning of the file. Any subsequent checks
> > >>> follow
> > >>> from the end of a successful match. Any captured variables are
> > >>> preserved, mostly because removing them would be hard.
> > >>
> > >>
> > >> The proposal is interesting, but I have doubts about the proposed
> > >> implementation. Wouldn't this make the semantics of some tests
> > >> very...
> > >> unusual? Consider this:
> > >>
> > >> CHECK: foo
> > >> CHECK-EVERYWHERE: bar
> > >> CHECK: baz
> > >>
> > >> Is this going to match:
> > >>
> > >> baz
> > >> foo
> > >> bar
> > >>
> > >> That's kind-of weird :-)
> > >
> > >
> > > I figured I should be more constructive :-)
> > > An alternative proposal without this problem (AFAICS) is to collect
> > > all
> > > "-EVERYWHERE" checks and run them separately in the end on the
> > > whole file.
> > > They're order-independent, so this seems safe. And oh while we're
> > > at it,
> > > let's call it CHECK-GLOBAL and also have CHECK-GLOBAL-NOT because
> > > that one
> > > would be really useful for rewriting some 'not grep' tests.
> > >
> > > Eli
> > >
> > >
> > _______________________________________________
> > llvm-commits mailing list
> > llvm-commits at cs.uiuc.edu <mailto:llvm-commits at cs.uiuc.edu>
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
> >
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu <mailto:llvm-commits at cs.uiuc.edu>
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>
>
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
More information about the llvm-commits
mailing list