<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Thanks for doing this! I am impressed that the test count is so high; what is that, around 2% of all tests? I wonder how many are typos (bad) versus genuinely unused (benign), but that would be tedious to determine.<o:p></o:p></p>
<p class="MsoNormal">Example of a typo: clang/test/AST/ast-dump-decl.c, MODULE vs MODULES<o:p></o:p></p>
<p class="MsoNormal">Example of unused: llvm/test/DebugInfo/X86/dbg-value-inlined-parameter.ll, specifies LINUX and DARWIN but neither is used.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’ll take this opportunity to mention that I’m planning a 10% project to look at the inverse question: What directives are in the test files but aren’t exercised by any FileCheck run? This is trickier as you need to collect data across
multiple runs, and tests run in parallel. Also it’s a bit ambiguous what “looks like” a directive, they aren’t really all intended to be directives. Should be fun.<o:p></o:p></p>
<p class="MsoNormal">If anyone is already delving into this, I’d be curious to hear about the approach.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">--paulr<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> llvm-dev <llvm-dev-bounces@lists.llvm.org> <b>On Behalf Of
</b>Mircea Trofin via llvm-dev<br>
<b>Sent:</b> Thursday, October 29, 2020 1:28 PM<br>
<b>To:</b> LLVM Dev <llvm-dev@lists.llvm.org><br>
<b>Subject:</b> [llvm-dev] [RFC] FileCheck: (dis)allowing unused prefixes<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Hello all,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">TL;DR; if you used FileCheck --check-prefixes and you missed (misspelled, for instance) one of the prefixes in your test, FileCheck silently ignores that and the test passes. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">1579 tests have this property. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><b>The details</b><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><b>=========</b><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Please refer to <a href="https://urldefense.com/v3/__https:/reviews.llvm.org/D90281__;!!JmoZiZGBv3RvKRSx!sQiWDnLaUAuuZLnJnmDYlIXvXwHr_j61zk-FLPNO_AhEEHEXZo_4yewxrqdv8DIPHA$">https://reviews.llvm.org/D90281</a> and the discussion there for
more details (make sure you open "older changes" for full context)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The problem is covered by the TL;DR;. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The proposal is to add an explicit flag to FileCheck, --allow-unused-prefixes, to indicate whether the current behavior is intended (for instance, jdoerfert contributed a scenario where that is the case). <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We want the default behavior to be 'strict', i.e. --allow-unused-prefixes=false. Doing that right now would lead to 1500 test failures.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">To get there (thanks, maskray, for suggestion), we propose we:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">* land D90281 where the flag is introduced, but is flipped to match today's behavior<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">* employ a 'busy beavers' approach, where test maintainers patch their tests:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> - either leveraging the flag, to explicitly indicate that unused prefixes is intended (i.e. add --allow-unused-patches=true); or<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> - fix the test (e.g. maybe there was a misspelling issue/omission/etc).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">A spreadsheet with the failing tests is available <a href="https://urldefense.com/v3/__https:/docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing__;!!JmoZiZGBv3RvKRSx!sQiWDnLaUAuuZLnJnmDYlIXvXwHr_j61zk-FLPNO_AhEEHEXZo_4yewxrqeR_nOyIg$">
here</a> [1].<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The request to the community members is to please sign up for their respective area in the spreadsheet, and then mark it completed when that's the case (yes/no in the respective column).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">When all the tests are fixed, we will then flip --allow-unused-prefixes to false by default.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Meanwhile, please consider leveraging the flag explicitly when you author new tests that use --check-prefixes. That can be then cleaned up easily after we switch to the 'strict' behavior.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">[1] <a href="https://urldefense.com/v3/__https:/docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing__;!!JmoZiZGBv3RvKRSx!sQiWDnLaUAuuZLnJnmDYlIXvXwHr_j61zk-FLPNO_AhEEHEXZo_4yewxrqeR_nOyIg$">https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>