[llvm-dev] FileCheck feature request- by default ignore IR-"headers"

[Robinson, Paul via llvm-dev]

re-add llvm-dev…

From: Robinson, Paul
Sent: Thursday, March 30, 2017 2:50 PM
To: 'Keane, Erich'; Reid Kleckner
Subject: RE: [llvm-dev] FileCheck feature request- by default ignore IR-"headers"

I think the most practical thing to do would be to make FileCheck reject test cases that consist only CHECK-NOT directives, unless a flag is passed to disable this behavior.

That might help a certain class of tests, but I'm not so sure that's such a common case.  It's about as easy to write a positive CHECK[-LABEL] that will unexpectedly match something that you didn't expect.  At least the test with only CHECK-NOT in it *will* fail if you get it wrong, which is certainly better than passing when it should fail.
So, I am reluctant to encourage yet another wart/hack on how FileCheck works.
--paulr

From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of Keane, Erich via llvm-dev
Sent: Thursday, March 30, 2017 11:18 AM
To: Reid Kleckner
Cc: llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
Subject: Re: [llvm-dev] FileCheck feature request- by default ignore IR-"headers"

Alright, I guess it isn’t just my pain then, it makes it feel better ☺  I think that proposed feature would be really nice, since it would encourage people to write tests that have a //CHECK: some-thing-after-header first!

From: Reid Kleckner [mailto:rnk at google.com]
Sent: Thursday, March 30, 2017 11:15 AM
To: Keane, Erich <erich.keane at intel.com<mailto:erich.keane at intel.com>>
Cc: llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
Subject: Re: [llvm-dev] FileCheck feature request- by default ignore IR-"headers"

Yeah, this is a documented shortcoming of FileCheck:
http://llvm.org/docs/TestingGuide.html#fragile-tests

I think the most practical thing to do would be to make FileCheck reject test cases that consist only CHECK-NOT directives, unless a flag is passed to disable this behavior.

On Thu, Mar 30, 2017 at 8:57 AM, Keane, Erich via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
Hi all-
I hope this is the right place for this.

Anyway, the primary usage for the utility "FileCheck" is to pattern match for specific values in a stream.  This is perfectly consistent and deterministic for the most part!  However, when validating clang's LLVM-IR generation, it is possible to make an invalid match against the top few generated lines (the LLVM-IR Headers), which are inconsistent.

For example (from my personal pain), we have a test: clang/test/CodeGen/debug-info-macro.c  that has a 'check-not' on the word "macros".  Unfortunately, my SVN workspace name was "ms_macros", which gave me a false match.  I'm a bit ashamed to say how long this took me to figure out :)

Anywya, to my feature request: to prevent someone else from going through this pain again, would it be possible to have FileCheck ignore these first 4 lines as default behavior?  I suspect there might be a few tests that would want to validate those headers, though I expect those are the vast minority, so I would think this should be opt in.

-Erich
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170330/68e10f3e/attachment.html>