<div dir="ltr"><div class="gmail_quote"><div class="gmail_attr">Hi Chris,<br></div><div dir="ltr" class="gmail_attr"><br></div><div dir="ltr" class="gmail_attr">On Thu, Jun 18, 2020 at 2:32 PM Chris Lattner <<a href="mailto:clattner@nondot.org">clattner@nondot.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div>Two questions:</div><div><br></div><div>1) Has anyone thought about adding a new “make check” configuration / envvar that enables more verbosity, which CI would enable but normal humans haven’t? The CI configs could be set to dump the whole file.</div></div></blockquote><div><br></div><div>FILECHECK_OPTS is an env var that permits this now. For example, I often do:<br></div><div><br></div><div>FILECHECK_OPTS='-dump-input=fail -vv -color' ninja check-all</div><div><br></div><div>Is that what you mean?<br></div><div><br></div><div>After the recent change, -dump-input=fail above is redundant, but maybe that default needs more discussion.<br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div><br></div><div>2) Instead of dumping the entire input by default, would it be reasonable to change the default “make check” to have FileCheck print the 10 lines before and after the mismatch? Most problems are close by the check failure. If you want to check extra fancy, dump the CHECK-LABEL region.</div><div><br></div><div>The combination of both seem like significant usability improvements for both CI and humans.</div></div></blockquote><div><br></div><div>Sounds reasonable to me.</div><div><br></div><div>Joel<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div><br></div><div>-Chris</div><div><br></div><br><div><br><blockquote type="cite"><div>On Jun 18, 2020, at 10:43 AM, Joel E. Denny via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:</div><br><div><div dir="ltr"><div dir="ltr"><div>As for the ordering issue... from the command line, I tend to set verbosity high, and so I pipe to a pager and search for "error:". I prefer errors before the input dump then. My point is that it should be configurable if we change the default ordering.<br></div><div><br></div><div>Joel<br></div><div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 18, 2020 at 1:13 PM Joel E. Denny <<a href="mailto:jdenny.ornl@gmail.com" target="_blank">jdenny.ornl@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div></div><div></div><div>Thanks for bringing this discussion to the list so we can see a broader set of opinions on what the default should be.</div><div><br></div><div>I like Johannes's suggestion that defaults should consider the new LLVM user. For that case, I think verbosity should be low, and diagnostics should mention how to increase verbosity.<br></div><div><br></div><div>For CI, I agree verbosity should be high because it's not easy for many LLVM contributors to try again with higher verbosity.<br></div><div><br></div><div>For my personal usage, I don't care what the default behavior is as long as behavior is easy to customize. FILECHECK_OPTS lets me do that per test suite run, or in my ~/.profile, or in my CI configs, etc.<br></div><div><br></div><div>Joel<br></div><div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 18, 2020 at 12:45 PM Sjoerd Meijer via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt">
I would guess that in a CI system the order doesn't matter much because you look at a webpage? I looked at some build bots today/yesterday that now also show this, and yeah, it's fine either way, I was guessing.<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt">
My primary use-case is usage in a terminal, and displaying the errors first followed by all input makes this pretty unusable.<br>
</div>
<div id="gmail-m_-6942248723404930883gmail-m_7993811440804990612gmail-m_-5473924934954905795appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-6942248723404930883gmail-m_7993811440804990612gmail-m_-5473924934954905795divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif"><b>From:</b> Chris Tetreault <<a href="mailto:ctetreau@quicinc.com" target="_blank">ctetreau@quicinc.com</a>><br>
<b>Sent:</b> 18 June 2020 17:34<br>
<b>To:</b> Sjoerd Meijer <<a href="mailto:Sjoerd.Meijer@arm.com" target="_blank">Sjoerd.Meijer@arm.com</a>><br>
<b>Cc:</b> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a> <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br>
<b>Subject:</b> RE: [llvm-dev] FileCheck</font>
<div> </div>
</div>
<div lang="EN-US">
<div><p>For anybody viewing these failures through some sort of CI system, showing the error first then the input file is more useful for the same reasons you mentioned. Personally, I rarely run filecheck by hand from the command prompt, so your
change would make my life worse. Granted, I’m just one person.</p><div> <br></div><p>The point I’m trying to make is that I don’t think it’s clear-cut which order is better, so maybe we shouldn’t change it. I think it might be fine to add an option to swap the order, but I’d be very sad if it started dumping to some random
file by default.</p><div> <br></div><p>Thanks,</p><p> Christopher Tetreault</p><div> <br></div>
<div>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in"><p><b>From:</b> llvm-dev <<a href="mailto:llvm-dev-bounces@lists.llvm.org" target="_blank">llvm-dev-bounces@lists.llvm.org</a>> <b>
On Behalf Of </b>Sjoerd Meijer via llvm-dev<br>
<b>Sent:</b> Thursday, June 18, 2020 9:16 AM<br>
<b>To:</b> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<b>Subject:</b> [EXT] [llvm-dev] FileCheck</p>
</div>
</div><div> <br></div>
<div><p><span style="font-size:12pt">Hello,</span></p>
</div>
<div><div><span style="font-size:12pt"> </span><br></div>
</div>
<div><p><span style="font-size:12pt">I am not sold on FileCheck's new behaviour. For failing tests in verbose mode, it first dump the actual error messages, followed by the annotated input file to FileCheck. The result is I can't
immediately see error messages if the input is more than just a few lines long, so I have to scroll all the way up to see the errors, then down again, etc.
</span></p>
</div>
<div><div><span style="font-size:12pt"> </span><br></div>
</div>
<div><p><span style="font-size:12pt">I do see some advantages of dumping the input to FileCheck, but an improvement for me would be:</span></p>
</div>
<div>
<ul type="disc">
<li><span style="font-size:12pt">to dump the input first, then followed by the error message, so that I can the errors first, and then decide to scroll up if I am interested to do so.</span></li><li><span style="font-size:12pt">dump it to a separate file (controlled with an option).</span></li></ul>
</div>
<div><p><span style="font-size:12pt">I am interested in changing the behaviour, because I think I find setting environment varibale "FILECHECK_OPTS="--dump-input never"" inconvenient. </span></p>
</div>
<div><div><span style="font-size:12pt"> </span><br></div>
</div>
<div><p><span style="font-size:12pt">My 2 pennies.</span></p>
</div>
<div><p><span style="font-size:12pt">Sjoerd.</span></p>
</div>
</div>
</div>
</div>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div></div>
</blockquote></div></div>
_______________________________________________<br>LLVM Developers mailing list<br><a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br><a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br></div></blockquote></div><br></div></blockquote></div></div>