[llvm-dev] Swallowing of input in FileCheck

[George Karpenkov via llvm-dev]

What about having an environment variable FILECHECKER_VERBOSE=1?
This would not require substitutions, and could be even set automatically by “lit” when launched with “-v”.
At least to me that would make debugging tests much easier.
> On Jul 7, 2017, at 3:05 PM, Daniel Dunbar <daniel_dunbar at apple.com> wrote:
> 
> 
>> On Jul 7, 2017, at 2:19 PM, Reid Kleckner <rnk at google.com <mailto:rnk at google.com>> wrote:
>> 
>> On Fri, Jul 7, 2017 at 1:20 PM, George Karpenkov via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> Thus, I propose modifying FileCheck default behavior to dump all swallowed output on stderr when the test has failed.
>> Would there be any objections to such a change?
>> 
>> Yes.
>>  
>> I understand the concern that log files might become unnecessarily large, but since it would only be done for failed
>> test I think the added readability would be worth it.
>> 
>> I disagree, it would be too much output. During development, it's pretty common to cause tens of tests to fail. I don't really want 10 entire assembly files dumped into my console during incremental development. Our test output is already long, and I wish it were shorter.
> 
> Could this be solved by having lit be intelligent about showing less output when there are large numbers of test failures (w/o other output), and truncating very large outputs?
> 
> I do think there are situations where having the output just show up by default locally could prevent needing to rerun a command, which is handy.
> 
>> I agree that this is a real problem when remote buildbots in different configurations get involved. Locally debugging FileCheck failures is easy, you just copy-paste the command like you said and pipe it to less. It's only a pain when you aren't sure if a failure on a bot will reproduce locally. So, I would be in favor of an option to lit that we enable on buildslaves that dumps the output. We already have a '\bFileCheck\b' substitution in lit. We'd just expand it to 'FileCheck --dump-on-failure' or something on bots.
> 
> This sounds reasonable to me, no matter what on the above question.
> 
>  - Daniel
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170707/3d530df9/attachment.html>