[llvm-commits] [PATCH] use FileCheck instead of grep (was Re: LIT cleanup phase #3 - remove usage of bash)

[Chandler Carruth]

On Wed, Apr 4, 2012 at 12:46 PM, Jay Foad <jay.foad at gmail.com> wrote:

> On 26 March 2012 09:50, Chandler Carruth <chandlerc at google.com> wrote:
> > Just thought I'd point out that removing the use of 'grep' from the test
> > suite has very significant benefits outside of any one platform.
> >
> > I regularly find regressions in LLVM where a 'not grep' pattern fails to
> > match because the comments change in the output, leaving bugs and other
> > problems undiagnosed. FileCheck-izing these tests is of significant long
> > term maintenance value IMO.
>
> Just to check I'm on the right track, does this patch look OK? It
> converts a couple of small tests to use FileCheck instead of grep.
> Tested with "make check".


There is nothing really wrong with these, but usually to be useful, you
have to go a bit further.

I think the main thing missing here is the idea of "anchoring" the
assertions that the test really cares about. You want to make sure that if
the assertion passes, it in fact passes for the right reason. Let's look at
simplified versions of the cases you're poking at:

1) Check that something is *not* present in the output. If we do this in
the standard way that the grep-invocations do this (not grep 'Foo') the
test can pass for completely inaccurate reasons: perhaps it simply produces
no output at all. Instead, we want to anchor the negative assertions with
some positive assertion that we're quite confident will be satisfied. In
the case of 2007-06-16-Funcname.ll, it's not clear that we have any good
'no-op' thing to check for. In that case, you can run the command in two
variants, one checking that Foo is present, and the second time that it is
gone.

1.B) While not relevant for the examples you mailed, this can get more
awkward when trying to check that something is not present inside a much
larger test case. There you may want to pin down exactly within what
*range* of things you are looking. This turns into a very common pattern:

CHECK: @my_test_function
CHECK-NOT: something_that_shouldnt_exist
CHECK: ret

2) Check that something *is* present in the output. This can get messed up
for strange reasons -- usually if some unrelated output in the test happens
to end up matching the output you're trying to assert. The typical way to
handle this is similar to 1.B, simply pin it down with some context. Often,
just the start is needed, unlike CHECK-NOT as the CHECK-NOT is active until
the EOF or the next CHECK.


Hope this helps, and hopefully not *too* overly verbose. ;]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20120404/d00ded64/attachment.html>