[llvm-dev] FileCheck
Dominik Montada via llvm-dev
llvm-dev at lists.llvm.org
Fri Jun 19 01:44:36 PDT 2020
Am 19.06.20 um 10:32 schrieb Mehdi AMINI via llvm-dev:
>
>
> On Fri, Jun 19, 2020 at 12:56 AM Sjoerd Meijer via llvm-dev
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>
> Sorry if I wasn't clear about my use case. In my daily dev work, I
> do many local "ninja check"s, or "llvm-lit" on a subdirectory as a
> quick(er) smoke test if I am making changes in that area (e.g.
> "llvm-lit ../llvm/test/CodeGen"). Nothing wrong here, as indeed
> nothing changed here. But in case of a test failure, I want to run
> just that test:
>
> bin/llvm-lit ../llvm/test/CodeGen/my_test.ll
>
> This only reports success/failure, and doesn't show any cause for
> failure , so I run it in verbose mode with:
>
> bin/llvm-lit -a ../llvm/test/CodeGen/my_test.ll
>
> In a terminal, the new default behaviour of FileCheck has become
> pretty unusable IMHO.
>
>
> I don't know how you proceed to debug FileCheck failures, but for me
> most of the time I'll have to figure out which "RUN" line fail and try
> to execute it manually and then remove the FileCheck pipe to get the
> raw input and then painfully tried to match the FileCheck error to the
> actual input. At least that's what I used to do before
> check-input=fail: this changed my relationship with debugging
> Filecheck failures!
>
> Matt mentioned that some codegen tests are very long: reducing the
> output by being contextual would likely help, but I'm also not
> convinced that this is a good practice to have so huge test in single
> files in the first place, it seems to me almost like anti-pattern and
> if it gets hard to read it is a good sign that it could benefit from
> some sharding.
Sometimes it's not really feasible to break up a test into multiple
smaller tests. For example, for GlobalISel tests are usually grouped per
stage and per instruction (i.e. legalize-merge-values.mir). Especially
those legalizer tests are often very extensive due to the large number
of type combinations. At least I wouldn't want to break up those tests
and clutter my test folder with tens of files for a single instruction
and a single stage.
I'm also not too happy with the current default behavior. Having the
error message printed last would already go a long way, but obviously
reducing the output in some contextual way would be best.
Dominik
>
>
> --
> Mehdi
>
>
>
> ------------------------------------------------------------------------
> *From:* Chris Tetreault <ctetreau at quicinc.com
> <mailto:ctetreau at quicinc.com>>
> *Sent:* 18 June 2020 20:49
> *To:* Joel E. Denny <jdenny.ornl at gmail.com
> <mailto:jdenny.ornl at gmail.com>>
> *Cc:* Sjoerd Meijer <Sjoerd.Meijer at arm.com
> <mailto:Sjoerd.Meijer at arm.com>>; llvm-dev at lists.llvm.org
> <mailto:llvm-dev at lists.llvm.org> <llvm-dev at lists.llvm.org
> <mailto:llvm-dev at lists.llvm.org>>
> *Subject:* RE: [llvm-dev] FileCheck
>
> Sjoerd specifically said “in verbose mode”, which I interpret to
> mean “when passing -v or -vv”. If we’re discussing the default
> behavior, then that’s a separate issue. Regardless, my other
> points stand independent of that.
>
> *From:* Joel E. Denny <jdenny.ornl at gmail.com
> <mailto:jdenny.ornl at gmail.com>>
> *Sent:* Thursday, June 18, 2020 12:43 PM
> *To:* Chris Tetreault <ctetreau at quicinc.com
> <mailto:ctetreau at quicinc.com>>
> *Cc:* Sjoerd Meijer <Sjoerd.Meijer at arm.com
> <mailto:Sjoerd.Meijer at arm.com>>; llvm-dev at lists.llvm.org
> <mailto:llvm-dev at lists.llvm.org>
> *Subject:* [EXT] Re: [llvm-dev] FileCheck
>
> On Thu, Jun 18, 2020 at 3:37 PM Chris Tetreault
> <ctetreau at quicinc.com <mailto:ctetreau at quicinc.com>> wrote:
>
> We’re talking about verbose output right? Verbose isn’t the
> default.
>
> I'm fairly certain the issue in this thread is just the verbosity
> of -dump-input=fail. Yes, -vv makes it even more verbose by
> annotating input lines with good matches, etc., but that's not
> part of the "new behaviour" Sjoerd meant, I believe.
>
> Joel
>
> *From:* Joel E. Denny <jdenny.ornl at gmail.com
> <mailto:jdenny.ornl at gmail.com>>
> *Sent:* Thursday, June 18, 2020 10:54 AM
> *To:* Chris Tetreault <ctetreau at quicinc.com
> <mailto:ctetreau at quicinc.com>>
> *Cc:* Sjoerd Meijer <Sjoerd.Meijer at arm.com
> <mailto:Sjoerd.Meijer at arm.com>>; llvm-dev at lists.llvm.org
> <mailto:llvm-dev at lists.llvm.org>
> *Subject:* [EXT] Re: [llvm-dev] FileCheck
>
> Hi Chris,
>
> On Thu, Jun 18, 2020 at 1:37 PM Chris Tetreault via llvm-dev
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>
> The thing I use normally only shows the first N lines by
> default (I don’t know off hand what N is). Honestly, I
> don’t feel very strongly about the specific order, but
> it’s not useful when somebody proposes something on the
> list, and nobody voices any dissent (choosing instead to
> silently oppose the change). My requests would be:
>
> 1. The order should be customizable via command line.
> 2. By default, it should not dump things to multiple
> locations. If I ask for verbose output, I want to get
> blasted with all the stuff.
> 3. The most important thing for me personally is to see
> the input to filecheck (I realize that this is in
> conflict with my earlier point. It’s early and I
> hadn’t had my coffee 😊 ). When I get a failure I want
> to be able to reproduce it in an IDE to use a
> debugger. Any change should not make this use case harder.
>
> Personally, I do not find the argument that the defaults
> should be setup to be best for newcomers to be very
> compelling; we are talking about changing the behavior of
> a non-default option after all.
>
> What do you mean by a "non-default option"? The default of
> -dump-input=never was recently changed to -dump-input=fail.
>
> Joel
>
> If just a bare filecheck invocation doesn’t tell a
> newcomer what they need to know, then they have to do
> filecheck -help or google the documentation anyways. At
> that point, they are going to customize it however they
> want. I assume anybody using filecheck to debug an issue
> is tech savvy enough to be able to configure the options,
> given reasonable documentation.
>
> Thanks,
>
> Christopher Tetreault
>
> *From:* Sjoerd Meijer <Sjoerd.Meijer at arm.com
> <mailto:Sjoerd.Meijer at arm.com>>
> *Sent:* Thursday, June 18, 2020 9:45 AM
> *To:* Chris Tetreault <ctetreau at quicinc.com
> <mailto:ctetreau at quicinc.com>>
> *Cc:* llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> *Subject:* [EXT] Re: [llvm-dev] FileCheck
>
> I would guess that in a CI system the order doesn't matter
> much because you look at a webpage? I looked at some build
> bots today/yesterday that now also show this, and yeah,
> it's fine either way, I was guessing.
>
> My primary use-case is usage in a terminal, and displaying
> the errors first followed by all input makes this pretty
> unusable.
>
> ------------------------------------------------------------------------
>
> *From:*Chris Tetreault <ctetreau at quicinc.com
> <mailto:ctetreau at quicinc.com>>
> *Sent:* 18 June 2020 17:34
> *To:* Sjoerd Meijer <Sjoerd.Meijer at arm.com
> <mailto:Sjoerd.Meijer at arm.com>>
> *Cc:* llvm-dev at lists.llvm.org
> <mailto:llvm-dev at lists.llvm.org><llvm-dev at lists.llvm.org
> <mailto:llvm-dev at lists.llvm.org>>
> *Subject:* RE: [llvm-dev] FileCheck
>
> For anybody viewing these failures through some sort of CI
> system, showing the error first then the input file is
> more useful for the same reasons you mentioned.
> Personally, I rarely run filecheck by hand from the
> command prompt, so your change would make my life worse.
> Granted, I’m just one person.
>
> The point I’m trying to make is that I don’t think it’s
> clear-cut which order is better, so maybe we shouldn’t
> change it. I think it might be fine to add an option to
> swap the order, but I’d be very sad if it started dumping
> to some random file by default.
>
> Thanks,
>
> Christopher Tetreault
>
> *From:* llvm-dev <llvm-dev-bounces at lists.llvm.org
> <mailto:llvm-dev-bounces at lists.llvm.org>> *On Behalf Of
> *Sjoerd Meijer via llvm-dev
> *Sent:* Thursday, June 18, 2020 9:16 AM
> *To:* llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> *Subject:* [EXT] [llvm-dev] FileCheck
>
> Hello,
>
> I am not sold on FileCheck's new behaviour. For failing
> tests in verbose mode, it first dump the actual error
> messages, followed by the annotated input file to
> FileCheck. The result is I can't immediately see error
> messages if the input is more than just a few lines long,
> so I have to scroll all the way up to see the errors, then
> down again, etc.
>
> I do see some advantages of dumping the input to
> FileCheck, but an improvement for me would be:
>
> * to dump the input first, then followed by the error
> message, so that I can the errors first, and then
> decide to scroll up if I am interested to do so.
> * dump it to a separate file (controlled with an option).
>
> I am interested in changing the behaviour, because I think
> I find setting environment varibale
> "FILECHECK_OPTS="--dump-input never"" inconvenient.
>
> My 2 pennies.
>
> Sjoerd.
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
--
----------------------------------------------------------------------
Dominik Montada Email: dominik.montada at hightec-rt.com
HighTec EDV-Systeme GmbH Phone: +49 681 92613 19
Europaallee 19 Fax: +49-681-92613-26
D-66113 Saarbrücken WWW: http://www.hightec-rt.com
Managing Director: Vera Strothmann
Register Court: Saarbrücken, HRB 10445, VAT ID: DE 138344222
This e-mail may contain confidential and/or privileged information. If
you are not the intended recipient please notify the sender immediately
and destroy this e-mail. Any unauthorised copying, disclosure or
distribution of the material in this e-mail is strictly forbidden.
---
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200619/0231f945/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 6822 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200619/0231f945/attachment-0001.bin>
More information about the llvm-dev
mailing list