[llvm-commits] [PATCH] A "very verbose" mode for FileCheck

[Eli Bendersky]

On Wed, Jan 16, 2013 at 11:43 AM, Dmitri Gribenko <gribozavr at gmail.com> wrote:
> On Wed, Jan 16, 2013 at 9:39 PM, Eli Bendersky <eliben at google.com> wrote:
>> On Wed, Jan 16, 2013 at 11:34 AM, David Blaikie <dblaikie at gmail.com> wrote:
>>> On Wed, Jan 16, 2013 at 11:30 AM, Dmitri Gribenko <gribozavr at gmail.com> wrote:
>>>> On Wed, Jan 16, 2013 at 8:47 PM, Dmitri Gribenko <gribozavr at gmail.com> wrote:
>>>>> On Wed, Jan 16, 2013 at 8:42 PM, David Blaikie <dblaikie at gmail.com> wrote:
>>>>>> On Wed, Jan 16, 2013 at 10:32 AM, Dmitri Gribenko <gribozavr at gmail.com> wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> When someone breaks a FileCheck-based test on some buildbot, sometimes
>>>>>>> it may not be obvious *why* did it fail.  If the failure can not be
>>>>>>> reproduced locally, it can be very hard to fix.
>>>>>>
>>>>>> Agreed.
>>>>>>
>>>>>>> I propose adding a "very verbose" mode to FileCheck.  In this mode
>>>>>>> FileCheck will dump the input file in case of failure.
>>>>>>
>>>>>> Why have a separate mode? The same problem of FileCheck failure
>>>>>> investigation exists even when developing locally - quite often after
>>>>>> a FileCheck failure the immediate next step is to run the command
>>>>>> without FileCheck just to get the output. Could we streamline this
>>>>>> scenario in general for both local and remote execution? (eg: have
>>>>>> FileCheck always dump a temp file & log the location)
>>>>>
>>>>> I agree that it is useful.  Will implement and post an updated version.
>>>>
>>>> Updated patch attached.  Now FileCheck prints:
>>>>
>>>> <all the usual messages>
>>>> Saving input file "<stdin>" to "/tmp/filecheck.txt-5PEf0q"
>>>> Input file "<stdin>" contains:
>>>> <...>
>>>
>>> Does this address my thought/desire that this should be the default
>>> behavior? (I assume people probably don't want the entire contents
>>> printed on each failure by default, so I guess the answer to my
>>> question is "no")
>>>
>>> If we had lit just log the file name on failure, we could improve the
>>> buildbot infrastructure to find/match this file name and generate a
>>> separate log file entry for it. This would allow us not to have a
>>> separate/different lit behavior on the bots while providing the full
>>> details to both local and buildbot use cases.
>>>
>>> (I'm not sure if anyone has objections to the idea of lit always
>>> dumping its input data, though - this could slow down test execution,
>>> for example)
>>> _________________________________
>>
>> Yes, the effect of dumping a (sometimes largish) file on each
>> invocation of FileCheck should be measured, since we run thousands
>> upon thousands of instances of FileCheck for each "make check-all".
>
> It is done only on errors, so I don't think we care about performance
> impact (if any).
>

Yes, this makes sense.

Eli