<div dir="ltr">FWIW - a -NOT only test looks especially problematic to me. The test would pass if the program produced no output, basically - which sounds overly loose (a shade less problematic than the "run this and don't check anything - not crashing is all we're testing for here" - which should have some positive test of the behavior that is expected other than crashing).<br><br>So I'd see such a constraint as potentially interesting on that basis - if it fixes this other issue as well, eh, helpful.<br><br>But I'm also OK with it being a thing we document, and every new contributor eventually learns once... it doesn't seem to have a lot of overhead.</div><br><div class="gmail_quote"><div dir="ltr">On Thu, Mar 30, 2017 at 2:50 PM Robinson, Paul via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple" class="gmail_msg">
<div class="m_-7118729956468737877WordSection1 gmail_msg">
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" class="gmail_msg">re-add llvm-dev…<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><a name="m_-7118729956468737877__MailEndCompose" class="gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></a></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt" class="gmail_msg">
<div class="gmail_msg">
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in" class="gmail_msg">
<p class="MsoNormal gmail_msg"><b class="gmail_msg"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class="gmail_msg">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class="gmail_msg"> Robinson, Paul
<br class="gmail_msg">
<b class="gmail_msg">Sent:</b> Thursday, March 30, 2017 2:50 PM<br class="gmail_msg">
<b class="gmail_msg">To:</b> 'Keane, Erich'; Reid Kleckner<br class="gmail_msg">
<b class="gmail_msg">Subject:</b> RE: [llvm-dev] FileCheck feature request- by default ignore IR-"headers"<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
</div>
</div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="gmail_msg"><div class="m_-7118729956468737877WordSection1 gmail_msg"><div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt" class="gmail_msg">
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg" style="margin-left:.5in">I think the most practical thing to do would be to make FileCheck reject test cases that consist only CHECK-NOT directives, unless a flag is passed to disable this behavior.<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
</div></div></div><div lang="EN-US" link="blue" vlink="purple" class="gmail_msg"><div class="m_-7118729956468737877WordSection1 gmail_msg"><div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt" class="gmail_msg"><p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" class="gmail_msg">That might help a certain class of tests, but I'm not so sure that's such a common case. It's about as easy to write a positive CHECK[-LABEL] that will unexpectedly
match something that you didn't expect. At least the test with only CHECK-NOT in it *will* fail if you get it wrong, which is certainly better than passing when it should fail.<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" class="gmail_msg">So, I am reluctant to encourage yet another wart/hack on how FileCheck works.<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" class="gmail_msg">--paulr<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt" class="gmail_msg">
<div class="gmail_msg">
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in" class="gmail_msg">
<p class="MsoNormal gmail_msg"><b class="gmail_msg"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class="gmail_msg">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class="gmail_msg"> llvm-dev [<a href="mailto:llvm-dev-bounces@lists.llvm.org" class="gmail_msg" target="_blank">mailto:llvm-dev-bounces@lists.llvm.org</a>]
<b class="gmail_msg">On Behalf Of </b>Keane, Erich via llvm-dev<br class="gmail_msg">
<b class="gmail_msg">Sent:</b> Thursday, March 30, 2017 11:18 AM<br class="gmail_msg">
<b class="gmail_msg">To:</b> Reid Kleckner</span></p></div></div></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="gmail_msg"><div class="m_-7118729956468737877WordSection1 gmail_msg"><div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt" class="gmail_msg"><div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt" class="gmail_msg"><div class="gmail_msg"><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in" class="gmail_msg"><p class="MsoNormal gmail_msg"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class="gmail_msg"><br class="gmail_msg">
<b class="gmail_msg">Cc:</b> <a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" target="_blank">llvm-dev@lists.llvm.org</a><br class="gmail_msg">
<b class="gmail_msg">Subject:</b> Re: [llvm-dev] FileCheck feature request- by default ignore IR-"headers"<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p></div></div></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="gmail_msg"><div class="m_-7118729956468737877WordSection1 gmail_msg"><div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt" class="gmail_msg"><div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt" class="gmail_msg">
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" class="gmail_msg">Alright, I guess it isn’t just my pain then, it makes it feel better
</span><span style="font-size:11.0pt;font-family:Wingdings;color:#1f497d" class="gmail_msg">J</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" class="gmail_msg"> I think that proposed feature would be really nice, since it would encourage people to write
tests that have a //CHECK: some-thing-after-header first!<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><b class="gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"" class="gmail_msg">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"" class="gmail_msg"> Reid Kleckner [<a href="mailto:rnk@google.com" class="gmail_msg" target="_blank">mailto:rnk@google.com</a>]
<br class="gmail_msg">
<b class="gmail_msg">Sent:</b> Thursday, March 30, 2017 11:15 AM<br class="gmail_msg">
<b class="gmail_msg">To:</b> Keane, Erich <<a href="mailto:erich.keane@intel.com" class="gmail_msg" target="_blank">erich.keane@intel.com</a>><br class="gmail_msg">
<b class="gmail_msg">Cc:</b> <a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" target="_blank">llvm-dev@lists.llvm.org</a><br class="gmail_msg">
<b class="gmail_msg">Subject:</b> Re: [llvm-dev] FileCheck feature request- by default ignore IR-"headers"<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
<div class="gmail_msg">
<div class="gmail_msg">
<p class="MsoNormal gmail_msg">Yeah, this is a documented shortcoming of FileCheck:<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><a href="http://llvm.org/docs/TestingGuide.html#fragile-tests" class="gmail_msg" target="_blank">http://llvm.org/docs/TestingGuide.html#fragile-tests</a><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
</div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg">I think the most practical thing to do would be to make FileCheck reject test cases that consist only CHECK-NOT directives, unless a flag is passed to disable this behavior.<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
</div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg">On Thu, Mar 30, 2017 at 8:57 AM, Keane, Erich via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt" class="gmail_msg">
<p class="MsoNormal gmail_msg">Hi all-<br class="gmail_msg">
I hope this is the right place for this.<br class="gmail_msg">
<br class="gmail_msg">
Anyway, the primary usage for the utility "FileCheck" is to pattern match for specific values in a stream. This is perfectly consistent and deterministic for the most part! However, when validating clang's LLVM-IR generation, it is possible to make an invalid
match against the top few generated lines (the LLVM-IR Headers), which are inconsistent.<br class="gmail_msg">
<br class="gmail_msg">
For example (from my personal pain), we have a test: clang/test/CodeGen/debug-info-macro.c that has a 'check-not' on the word "macros". Unfortunately, my SVN workspace name was "ms_macros", which gave me a false match. I'm a bit ashamed to say how long this
took me to figure out :)<br class="gmail_msg">
<br class="gmail_msg">
Anywya, to my feature request: to prevent someone else from going through this pain again, would it be possible to have FileCheck ignore these first 4 lines as default behavior? I suspect there might be a few tests that would want to validate those headers,
though I expect those are the vast minority, so I would think this should be opt in.<br class="gmail_msg">
<br class="gmail_msg">
-Erich<br class="gmail_msg">
_______________________________________________<br class="gmail_msg">
LLVM Developers mailing list<br class="gmail_msg">
<a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" target="_blank">llvm-dev@lists.llvm.org</a><br class="gmail_msg">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" class="gmail_msg" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</blockquote>
</div>
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
</div>
</div></div></div></div>
_______________________________________________<br class="gmail_msg">
LLVM Developers mailing list<br class="gmail_msg">
<a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" target="_blank">llvm-dev@lists.llvm.org</a><br class="gmail_msg">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="gmail_msg">
</blockquote></div>