[llvm-dev] FileCheck
Mehdi AMINI via llvm-dev
llvm-dev at lists.llvm.org
Fri Jun 19 02:56:05 PDT 2020
On Fri, Jun 19, 2020 at 1:59 AM Sjoerd Meijer <Sjoerd.Meijer at arm.com> wrote:
> Ah, I forgot to ask this:
>
> > At least that's what I used to do before check-input=fail: this changed
> my relationship with debugging Filecheck failures!
>
> What do you mean by this? Curious if I am missing a neat trick here?
>
Nothing super tricky.
For example I swapped two lines in a test:
// CHECK: [[VAL_16:%.*]] = muli [[NEW_I1]], [[C13]] : index
// CHECK: [[I0:%.*]] = remi_signed [[NEW_I0]], [[C2]] : index
And ran it with `./bin/llvm-lit -sv
../mlir/test/Transforms/parallel-loop-collapsing.mlir`.
The output is unsurprisingly:
llvm-project/mlir/test/Transforms/parallel-loop-collapsing.mlir:40:11:
error: CHECK: expected string not found in input
// CHECK: [[I0:%.*]] = remi_signed [[NEW_I0]], [[C2]] : index
^
<stdin>:18:2: note: scanning from here
%2 = addi %1, %c12 : index
^
There isn't enough information to do much here, while dump-input=fail adds:
Full input was:
<<<<<<
1:
2:
3: module {
4: func @parallel_many_dims() {
5: %c6 = constant 6 : index
6: %c7 = constant 7 : index
7: %c9 = constant 9 : index
8: %c10 = constant 10 : index
9: %c12 = constant 12 : index
10: %c13 = constant 13 : index
11: %c3 = constant 3 : index
12: %c0 = constant 0 : index
13: %c1 = constant 1 : index
14: %c2 = constant 2 : index
15: scf.parallel (%arg0, %arg1, %arg2) = (%c0, %c0, %c0) to (%c2,
%c1, %c1) step (%c1, %c1, %c1) {
16: %0 = remi_signed %arg0, %c2 : index
17: %1 = muli %arg1, %c13 : index
18: %2 = addi %1, %c12 : index
check:40 X~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
19: %3 = muli %arg0, %c10 : index
check:40 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
20: %4 = addi %3, %c9 : index
check:40 ~~~~~~~~~~~~~~~~~~~~~~~~~~
21: %5 = muli %arg2, %c7 : index
check:40 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
22: %6 = addi %5, %c6 : index
check:40 ~~~~~~~~~~~~~~~~~~~~~~~~~~
23: %7 = "magic.op"(%0, %c3, %6, %4, %2) : (index, index, index,
index, index) -> index
check:40
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
24: scf.yield
check:40 ~~~~~~~~~~
25: }
check:40 ~~
26: return
check:40 ~~~~~~~
27: }
check:40 ~~
28: }
check:40 ~
>>>>>>
Which is all I need to understand where/how FileCheck got stuck.
--
Mehdi
> ------------------------------
> *From:* Sjoerd Meijer <Sjoerd.Meijer at arm.com>
> *Sent:* 19 June 2020 09:56
> *To:* Mehdi AMINI <joker.eph at gmail.com>
> *Cc:* Chris Tetreault <ctetreau at quicinc.com>; Joel E. Denny <
> jdenny.ornl at gmail.com>; llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org>
> *Subject:* Re: [llvm-dev] FileCheck
>
> > I don't know how you proceed to debug FileCheck failures, but for me
> most of the time I'll have to figure out which "RUN" line fail and try to
> execute it manually and then remove the FileCheck pipe to get the raw input
> and then painfully tried to match the FileCheck error to the actual input.
>
> Yeah, not very different from what you described here. If I 'm creating or
> editing a test file, just reading the error message catches a lot of cases
> for me. But yes, if it is more complicated I do exactly what you said,
> which is definitely a bit labour intensive, not ideal, and that's why I
> also think dumping the input to FileCheck is very useful, but for me not in
> its current form. For that, first, I think we only need to dump the
> context, and not the entire file, as also remarked earlier in this thread.
> Second, we can discuss the order of the output (or if is dumped to a
> file/stdout/stderr).
>
> > Matt mentioned that some codegen tests are very long: reducing the
> output by being contextual would likely help, but I'm also not convinced
> that this is a good practice to have so huge test in single files in the
> first place, it seems to me almost like anti-pattern and if it gets hard to
> read it is a good sign that it could benefit from some sharding.
>
> While I agree with this, I think the structure of the tests is a separate
> discussion. That is, I think you can have 1 test-case in 1 file, and the
> current behaviour is already inconvenient, so don't see that as a solution.
>
> Cheers.
>
>
> ------------------------------
> *From:* Mehdi AMINI <joker.eph at gmail.com>
> *Sent:* 19 June 2020 09:32
> *To:* Sjoerd Meijer <Sjoerd.Meijer at arm.com>
> *Cc:* Chris Tetreault <ctetreau at quicinc.com>; Joel E. Denny <
> jdenny.ornl at gmail.com>; llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org>
> *Subject:* Re: [llvm-dev] FileCheck
>
>
>
> On Fri, Jun 19, 2020 at 12:56 AM Sjoerd Meijer via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Sorry if I wasn't clear about my use case. In my daily dev work, I do many
> local "ninja check"s, or "llvm-lit" on a subdirectory as a quick(er) smoke
> test if I am making changes in that area (e.g. "llvm-lit
> ../llvm/test/CodeGen"). Nothing wrong here, as indeed nothing changed here.
> But in case of a test failure, I want to run just that test:
>
> bin/llvm-lit ../llvm/test/CodeGen/my_test.ll
>
> This only reports success/failure, and doesn't show any cause for failure
> , so I run it in verbose mode with:
>
> bin/llvm-lit -a ../llvm/test/CodeGen/my_test.ll
>
> In a terminal, the new default behaviour of FileCheck has become pretty
> unusable IMHO.
>
>
> I don't know how you proceed to debug FileCheck failures, but for me most
> of the time I'll have to figure out which "RUN" line fail and try to
> execute it manually and then remove the FileCheck pipe to get the raw input
> and then painfully tried to match the FileCheck error to the actual input.
> At least that's what I used to do before check-input=fail: this changed my
> relationship with debugging Filecheck failures!
>
> Matt mentioned that some codegen tests are very long: reducing the output
> by being contextual would likely help, but I'm also not convinced that this
> is a good practice to have so huge test in single files in the first place,
> it seems to me almost like anti-pattern and if it gets hard to read it is a
> good sign that it could benefit from some sharding.
>
>
> --
> Mehdi
>
>
>
>
>
> ------------------------------
> *From:* Chris Tetreault <ctetreau at quicinc.com>
> *Sent:* 18 June 2020 20:49
> *To:* Joel E. Denny <jdenny.ornl at gmail.com>
> *Cc:* Sjoerd Meijer <Sjoerd.Meijer at arm.com>; llvm-dev at lists.llvm.org <
> llvm-dev at lists.llvm.org>
> *Subject:* RE: [llvm-dev] FileCheck
>
>
> Sjoerd specifically said “in verbose mode”, which I interpret to mean
> “when passing -v or -vv”. If we’re discussing the default behavior, then
> that’s a separate issue. Regardless, my other points stand independent of
> that.
>
>
>
> *From:* Joel E. Denny <jdenny.ornl at gmail.com>
> *Sent:* Thursday, June 18, 2020 12:43 PM
> *To:* Chris Tetreault <ctetreau at quicinc.com>
> *Cc:* Sjoerd Meijer <Sjoerd.Meijer at arm.com>; llvm-dev at lists.llvm.org
> *Subject:* [EXT] Re: [llvm-dev] FileCheck
>
>
>
> On Thu, Jun 18, 2020 at 3:37 PM Chris Tetreault <ctetreau at quicinc.com>
> wrote:
>
> We’re talking about verbose output right? Verbose isn’t the default.
>
>
>
> I'm fairly certain the issue in this thread is just the verbosity of
> -dump-input=fail. Yes, -vv makes it even more verbose by annotating input
> lines with good matches, etc., but that's not part of the "new behaviour"
> Sjoerd meant, I believe.
>
>
>
> Joel
>
>
>
>
>
> *From:* Joel E. Denny <jdenny.ornl at gmail.com>
> *Sent:* Thursday, June 18, 2020 10:54 AM
> *To:* Chris Tetreault <ctetreau at quicinc.com>
> *Cc:* Sjoerd Meijer <Sjoerd.Meijer at arm.com>; llvm-dev at lists.llvm.org
> *Subject:* [EXT] Re: [llvm-dev] FileCheck
>
>
>
> Hi Chris,
>
>
>
> On Thu, Jun 18, 2020 at 1:37 PM Chris Tetreault via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> The thing I use normally only shows the first N lines by default (I don’t
> know off hand what N is). Honestly, I don’t feel very strongly about the
> specific order, but it’s not useful when somebody proposes something on the
> list, and nobody voices any dissent (choosing instead to silently oppose
> the change). My requests would be:
>
>
>
> 1. The order should be customizable via command line.
> 2. By default, it should not dump things to multiple locations. If I
> ask for verbose output, I want to get blasted with all the stuff.
> 3. The most important thing for me personally is to see the input to
> filecheck (I realize that this is in conflict with my earlier point. It’s
> early and I hadn’t had my coffee 😊 ). When I get a failure I want to
> be able to reproduce it in an IDE to use a debugger. Any change should not
> make this use case harder.
>
>
>
> Personally, I do not find the argument that the defaults should be setup
> to be best for newcomers to be very compelling; we are talking about
> changing the behavior of a non-default option after all.
>
>
>
> What do you mean by a "non-default option"? The default of
> -dump-input=never was recently changed to -dump-input=fail.
>
>
>
> Joel
>
>
>
> If just a bare filecheck invocation doesn’t tell a newcomer what they need
> to know, then they have to do filecheck -help or google the documentation
> anyways. At that point, they are going to customize it however they want. I
> assume anybody using filecheck to debug an issue is tech savvy enough to be
> able to configure the options, given reasonable documentation.
>
>
>
> Thanks,
>
> Christopher Tetreault
>
>
>
> *From:* Sjoerd Meijer <Sjoerd.Meijer at arm.com>
> *Sent:* Thursday, June 18, 2020 9:45 AM
> *To:* Chris Tetreault <ctetreau at quicinc.com>
> *Cc:* llvm-dev at lists.llvm.org
> *Subject:* [EXT] Re: [llvm-dev] FileCheck
>
>
>
>
>
> I would guess that in a CI system the order doesn't matter much because
> you look at a webpage? I looked at some build bots today/yesterday that now
> also show this, and yeah, it's fine either way, I was guessing.
>
>
>
> My primary use-case is usage in a terminal, and displaying the errors
> first followed by all input makes this pretty unusable.
> ------------------------------
>
> *From:* Chris Tetreault <ctetreau at quicinc.com>
> *Sent:* 18 June 2020 17:34
> *To:* Sjoerd Meijer <Sjoerd.Meijer at arm.com>
> *Cc:* llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org>
> *Subject:* RE: [llvm-dev] FileCheck
>
>
>
> For anybody viewing these failures through some sort of CI system, showing
> the error first then the input file is more useful for the same reasons you
> mentioned. Personally, I rarely run filecheck by hand from the command
> prompt, so your change would make my life worse. Granted, I’m just one
> person.
>
>
>
> The point I’m trying to make is that I don’t think it’s clear-cut which
> order is better, so maybe we shouldn’t change it. I think it might be fine
> to add an option to swap the order, but I’d be very sad if it started
> dumping to some random file by default.
>
>
>
> Thanks,
>
> Christopher Tetreault
>
>
>
> *From:* llvm-dev <llvm-dev-bounces at lists.llvm.org> *On Behalf Of *Sjoerd
> Meijer via llvm-dev
> *Sent:* Thursday, June 18, 2020 9:16 AM
> *To:* llvm-dev at lists.llvm.org
> *Subject:* [EXT] [llvm-dev] FileCheck
>
>
>
> Hello,
>
>
>
> I am not sold on FileCheck's new behaviour. For failing tests in verbose
> mode, it first dump the actual error messages, followed by the annotated
> input file to FileCheck. The result is I can't immediately see error
> messages if the input is more than just a few lines long, so I have to
> scroll all the way up to see the errors, then down again, etc.
>
>
>
> I do see some advantages of dumping the input to FileCheck, but an
> improvement for me would be:
>
> - to dump the input first, then followed by the error message, so that
> I can the errors first, and then decide to scroll up if I am interested to
> do so.
> - dump it to a separate file (controlled with an option).
>
> I am interested in changing the behaviour, because I think I find setting
> environment varibale "FILECHECK_OPTS="--dump-input never"" inconvenient.
>
>
>
> My 2 pennies.
>
> Sjoerd.
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200619/6440fcd2/attachment.html>
More information about the llvm-dev
mailing list