[llvm-dev] FileCheck feature request- by default ignore IR-"headers"

[David Blaikie via llvm-dev]

FWIW - a -NOT only test looks especially problematic to me. The test would
pass if the program produced no output, basically - which sounds overly
loose (a shade less problematic than the "run this and don't check anything
- not crashing is all we're testing for here" - which should have some
positive test of the behavior that is expected other than crashing).

So I'd see such a constraint as potentially interesting on that basis - if
it fixes this other issue as well, eh, helpful.

But I'm also OK with it being a thing we document, and every new
contributor eventually learns once... it doesn't seem to have a lot of
overhead.

On Thu, Mar 30, 2017 at 2:50 PM Robinson, Paul via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> re-add llvm-dev…
>
>
>
> *From:* Robinson, Paul
> *Sent:* Thursday, March 30, 2017 2:50 PM
> *To:* 'Keane, Erich'; Reid Kleckner
> *Subject:* RE: [llvm-dev] FileCheck feature request- by default ignore
> IR-"headers"
>
>
>
> I think the most practical thing to do would be to make FileCheck reject
> test cases that consist only CHECK-NOT directives, unless a flag is passed
> to disable this behavior.
>
>
>
> That might help a certain class of tests, but I'm not so sure that's such
> a common case.  It's about as easy to write a positive CHECK[-LABEL] that
> will unexpectedly match something that you didn't expect.  At least the
> test with only CHECK-NOT in it *will* fail if you get it wrong, which is
> certainly better than passing when it should fail.
>
> So, I am reluctant to encourage yet another wart/hack on how FileCheck
> works.
>
> --paulr
>
>
>
> *From:* llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org
> <llvm-dev-bounces at lists.llvm.org>] *On Behalf Of *Keane, Erich via
> llvm-dev
> *Sent:* Thursday, March 30, 2017 11:18 AM
> *To:* Reid Kleckner
>
>
> *Cc:* llvm-dev at lists.llvm.org
> *Subject:* Re: [llvm-dev] FileCheck feature request- by default ignore
> IR-"headers"
>
>
>
> Alright, I guess it isn’t just my pain then, it makes it feel better J  I
> think that proposed feature would be really nice, since it would encourage
> people to write tests that have a //CHECK: some-thing-after-header first!
>
>
>
> *From:* Reid Kleckner [mailto:rnk at google.com <rnk at google.com>]
> *Sent:* Thursday, March 30, 2017 11:15 AM
> *To:* Keane, Erich <erich.keane at intel.com>
> *Cc:* llvm-dev at lists.llvm.org
> *Subject:* Re: [llvm-dev] FileCheck feature request- by default ignore
> IR-"headers"
>
>
>
> Yeah, this is a documented shortcoming of FileCheck:
>
> http://llvm.org/docs/TestingGuide.html#fragile-tests
>
>
>
> I think the most practical thing to do would be to make FileCheck reject
> test cases that consist only CHECK-NOT directives, unless a flag is passed
> to disable this behavior.
>
>
>
> On Thu, Mar 30, 2017 at 8:57 AM, Keane, Erich via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Hi all-
> I hope this is the right place for this.
>
> Anyway, the primary usage for the utility "FileCheck" is to pattern match
> for specific values in a stream.  This is perfectly consistent and
> deterministic for the most part!  However, when validating clang's LLVM-IR
> generation, it is possible to make an invalid match against the top few
> generated lines (the LLVM-IR Headers), which are inconsistent.
>
> For example (from my personal pain), we have a test:
> clang/test/CodeGen/debug-info-macro.c  that has a 'check-not' on the word
> "macros".  Unfortunately, my SVN workspace name was "ms_macros", which gave
> me a false match.  I'm a bit ashamed to say how long this took me to figure
> out :)
>
> Anywya, to my feature request: to prevent someone else from going through
> this pain again, would it be possible to have FileCheck ignore these first
> 4 lines as default behavior?  I suspect there might be a few tests that
> would want to validate those headers, though I expect those are the vast
> minority, so I would think this should be opt in.
>
> -Erich
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170331/22f022a3/attachment.html>