[llvm-commits] [PATCH] A "very verbose" mode for FileCheck

[David Blaikie]

On Wed, Jan 16, 2013 at 11:44 AM, Eli Bendersky <eliben at google.com> wrote:
> On Wed, Jan 16, 2013 at 11:43 AM, Dmitri Gribenko <gribozavr at gmail.com> wrote:
>> On Wed, Jan 16, 2013 at 9:39 PM, Eli Bendersky <eliben at google.com> wrote:
>>> On Wed, Jan 16, 2013 at 11:34 AM, David Blaikie <dblaikie at gmail.com> wrote:
>>>> On Wed, Jan 16, 2013 at 11:30 AM, Dmitri Gribenko <gribozavr at gmail.com> wrote:
>>>>> On Wed, Jan 16, 2013 at 8:47 PM, Dmitri Gribenko <gribozavr at gmail.com> wrote:
>>>>>> On Wed, Jan 16, 2013 at 8:42 PM, David Blaikie <dblaikie at gmail.com> wrote:
>>>>>>> On Wed, Jan 16, 2013 at 10:32 AM, Dmitri Gribenko <gribozavr at gmail.com> wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> When someone breaks a FileCheck-based test on some buildbot, sometimes
>>>>>>>> it may not be obvious *why* did it fail.  If the failure can not be
>>>>>>>> reproduced locally, it can be very hard to fix.
>>>>>>>
>>>>>>> Agreed.
>>>>>>>
>>>>>>>> I propose adding a "very verbose" mode to FileCheck.  In this mode
>>>>>>>> FileCheck will dump the input file in case of failure.
>>>>>>>
>>>>>>> Why have a separate mode? The same problem of FileCheck failure
>>>>>>> investigation exists even when developing locally - quite often after
>>>>>>> a FileCheck failure the immediate next step is to run the command
>>>>>>> without FileCheck just to get the output. Could we streamline this
>>>>>>> scenario in general for both local and remote execution? (eg: have
>>>>>>> FileCheck always dump a temp file & log the location)
>>>>>>
>>>>>> I agree that it is useful.  Will implement and post an updated version.
>>>>>
>>>>> Updated patch attached.  Now FileCheck prints:
>>>>>
>>>>> <all the usual messages>
>>>>> Saving input file "<stdin>" to "/tmp/filecheck.txt-5PEf0q"
>>>>> Input file "<stdin>" contains:
>>>>> <...>
>>>>
>>>> Does this address my thought/desire that this should be the default
>>>> behavior? (I assume people probably don't want the entire contents
>>>> printed on each failure by default, so I guess the answer to my
>>>> question is "no")
>>>>
>>>> If we had lit just log the file name on failure, we could improve the
>>>> buildbot infrastructure to find/match this file name and generate a
>>>> separate log file entry for it. This would allow us not to have a
>>>> separate/different lit behavior on the bots while providing the full
>>>> details to both local and buildbot use cases.
>>>>
>>>> (I'm not sure if anyone has objections to the idea of lit always
>>>> dumping its input data, though - this could slow down test execution,
>>>> for example)
>>>> _________________________________
>>>
>>> Yes, the effect of dumping a (sometimes largish) file on each
>>> invocation of FileCheck should be measured, since we run thousands
>>> upon thousands of instances of FileCheck for each "make check-all".
>>
>> It is done only on errors, so I don't think we care about performance
>> impact (if any).

How does this happen only on error? Is FileCheck already buffering the
entire contents of the input anyway?

Well, in any case, that's fine by me.

But leaves me with the question: why bother with the option to print
to stderr at all? We can just improve the buildbot configuration to
log these files alongside the failure log. (ensuring that they'll be
complete/standalone/exact replicas of the input & you won't have to
figure out where the file input begins/ends and the FileCheck/other
log data is)

>>
>
> Yes, this makes sense.