[llvm-dev] FileCheck

Thu Jun 18 11:40:03 PDT 2020

Hi Chris,

On Thu, Jun 18, 2020 at 2:32 PM Chris Lattner <clattner at nondot.org> wrote:

> Two questions:
>
> 1) Has anyone thought about adding a new “make check” configuration /
> envvar that enables more verbosity, which CI would enable but normal humans
> haven’t?  The CI configs could be set to dump the whole file.
>

FILECHECK_OPTS is an env var that permits this now.  For example, I often
do:

FILECHECK_OPTS='-dump-input=fail -vv -color' ninja check-all

Is that what you mean?

After the recent change, -dump-input=fail above is redundant, but maybe
that default needs more discussion.

> 2) Instead of dumping the entire input by default, would it be reasonable
> to change the default “make check” to have  FileCheck print the 10 lines
> before and after the mismatch?  Most problems are close by the check
> failure.  If you want to check extra fancy, dump the CHECK-LABEL region.
>
> The combination of both seem like significant usability improvements for
> both CI and humans.
>

Sounds reasonable to me.

Joel

>
> -Chris
>
>
>
> On Jun 18, 2020, at 10:43 AM, Joel E. Denny via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> As for the ordering issue... from the command line, I tend to set
> verbosity high, and so I pipe to a pager and search for "error:".  I prefer
> errors before the input dump then.  My point is that it should be
> configurable if we change the default ordering.
>
> Joel
>
> On Thu, Jun 18, 2020 at 1:13 PM Joel E. Denny <jdenny.ornl at gmail.com>
> wrote:
>
>> Thanks for bringing this discussion to the list so we can see a broader
>> set of opinions on what the default should be.
>>
>> I like Johannes's suggestion that defaults should consider the new LLVM
>> user.  For that case, I think verbosity should be low, and diagnostics
>> should mention how to increase verbosity.
>>
>> For CI, I agree verbosity should be high because it's not easy for many
>> LLVM contributors to try again with higher verbosity.
>>
>> For my personal usage, I don't care what the default behavior is as long
>> as behavior is easy to customize.  FILECHECK_OPTS lets me do that per test
>> suite run, or in my ~/.profile, or in my CI configs, etc.
>>
>> Joel
>>
>> On Thu, Jun 18, 2020 at 12:45 PM Sjoerd Meijer via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>>
>>> I would guess that in a CI system the order doesn't matter much because
>>> you look at a webpage? I looked at some build bots today/yesterday that now
>>> also show this, and yeah, it's fine either way, I was guessing.
>>>
>>> My primary use-case is usage in a terminal, and displaying the errors
>>> first followed by all input makes this pretty unusable.
>>> ------------------------------
>>> *From:* Chris Tetreault <ctetreau at quicinc.com>
>>> *Sent:* 18 June 2020 17:34
>>> *To:* Sjoerd Meijer <Sjoerd.Meijer at arm.com>
>>> *Cc:* llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org>
>>> *Subject:* RE: [llvm-dev] FileCheck
>>>
>>>
>>> For anybody viewing these failures through some sort of CI system,
>>> showing the error first then the input file is more useful for the same
>>> reasons you mentioned. Personally, I rarely run filecheck by hand from the
>>> command prompt, so your change would make my life worse. Granted, I’m just
>>> one person.
>>>
>>>
>>> The point I’m trying to make is that I don’t think it’s clear-cut which
>>> order is better, so maybe we shouldn’t change it. I think it might be fine
>>> to add an option to swap the order, but I’d be very sad if it started
>>> dumping to some random file by default.
>>>
>>>
>>> Thanks,
>>>
>>>    Christopher Tetreault
>>>
>>>
>>> *From:* llvm-dev <llvm-dev-bounces at lists.llvm.org> * On Behalf Of *Sjoerd
>>> Meijer via llvm-dev
>>> *Sent:* Thursday, June 18, 2020 9:16 AM
>>> *To:* llvm-dev at lists.llvm.org
>>> *Subject:* [EXT] [llvm-dev] FileCheck
>>>
>>>
>>> Hello,
>>>
>>>
>>> I am not sold on FileCheck's new behaviour. For failing tests in verbose
>>> mode, it first dump the actual error messages, followed by the annotated
>>> input file to FileCheck. The result is I can't immediately see error
>>> messages if the input is more than just a few lines long, so I have to
>>> scroll all the way up to see the errors, then down again, etc.
>>>
>>>
>>> I do see some advantages of dumping the input to FileCheck, but an
>>> improvement for me would be:
>>>
>>>    - to dump the input first, then followed by the error message, so
>>>    that I can the errors first, and then decide to scroll up if I am
>>>    interested to do so.
>>>    - dump it to a separate file (controlled with an option).
>>>
>>> I am interested in changing the behaviour, because I think I find
>>> setting environment varibale "FILECHECK_OPTS="--dump-input never""
>>> inconvenient.
>>>
>>>
>>> My 2 pennies.
>>>
>>> Sjoerd.
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200618/f8784fad/attachment.html>