[llvm-commits] [PATCH] A "very verbose" mode for FileCheck

[Eli Bendersky]

On Wed, Jan 16, 2013 at 11:34 AM, David Blaikie <dblaikie at gmail.com> wrote:
> On Wed, Jan 16, 2013 at 11:30 AM, Dmitri Gribenko <gribozavr at gmail.com> wrote:
>> On Wed, Jan 16, 2013 at 8:47 PM, Dmitri Gribenko <gribozavr at gmail.com> wrote:
>>> On Wed, Jan 16, 2013 at 8:42 PM, David Blaikie <dblaikie at gmail.com> wrote:
>>>> On Wed, Jan 16, 2013 at 10:32 AM, Dmitri Gribenko <gribozavr at gmail.com> wrote:
>>>>> Hello,
>>>>>
>>>>> When someone breaks a FileCheck-based test on some buildbot, sometimes
>>>>> it may not be obvious *why* did it fail.  If the failure can not be
>>>>> reproduced locally, it can be very hard to fix.
>>>>
>>>> Agreed.
>>>>
>>>>> I propose adding a "very verbose" mode to FileCheck.  In this mode
>>>>> FileCheck will dump the input file in case of failure.
>>>>
>>>> Why have a separate mode? The same problem of FileCheck failure
>>>> investigation exists even when developing locally - quite often after
>>>> a FileCheck failure the immediate next step is to run the command
>>>> without FileCheck just to get the output. Could we streamline this
>>>> scenario in general for both local and remote execution? (eg: have
>>>> FileCheck always dump a temp file & log the location)
>>>
>>> I agree that it is useful.  Will implement and post an updated version.
>>
>> Updated patch attached.  Now FileCheck prints:
>>
>> <all the usual messages>
>> Saving input file "<stdin>" to "/tmp/filecheck.txt-5PEf0q"
>> Input file "<stdin>" contains:
>> <...>
>
> Does this address my thought/desire that this should be the default
> behavior? (I assume people probably don't want the entire contents
> printed on each failure by default, so I guess the answer to my
> question is "no")
>
> If we had lit just log the file name on failure, we could improve the
> buildbot infrastructure to find/match this file name and generate a
> separate log file entry for it. This would allow us not to have a
> separate/different lit behavior on the bots while providing the full
> details to both local and buildbot use cases.
>
> (I'm not sure if anyone has objections to the idea of lit always
> dumping its input data, though - this could slow down test execution,
> for example)
> _________________________________

Yes, the effect of dumping a (sometimes largish) file on each
invocation of FileCheck should be measured, since we run thousands
upon thousands of instances of FileCheck for each "make check-all".

Eli